Abstract:The objective in a traditional reinforcement learning (RL) problem is to find a policy that optimizes the expected value of a performance metric such as the infinite-horizon cumulative discounted or long-run average cost/reward. In practice, optimizing the expected value alone may not be satisfactory, in that it may be desirable to incorporate the notion of risk into the optimization problem formulation, either in the objective or as a constraint. Various risk measures have been proposed in the literature, e.g., exponential utility, variance, percentile performance, chance constraints, value at risk (quantile), conditional value-at-risk, prospect theory and its later enhancement, cumulative prospect theory. In this book, we consider risk-sensitive RL in two settings: one where the goal is to find a policy that optimizes the usual expected value objective while ensuring that a risk constraint is satisfied, and the other where the risk measure is the objective. We survey some of the recent work in this area specifically where policy gradient search is the solution approach. In the first risk-sensitive RL setting, we cover popular risk measures based on variance, conditional value-at-risk, and chance constraints, and present a template for policy gradient-based risk-sensitive RL algorithms using a Lagrangian formulation. For the setting where risk is incorporated directly into the objective function, we consider an exponential utility formulation, cumulative prospect theory, and coherent risk measures. This non-exhaustive survey aims to give a flavor of the challenges involved in solving risk-sensitive RL problems using policy gradient methods, as well as outlining some potential future research directions.

Entropic Risk Measure in Policy Search

Safe Sim-to-Real Robot Exploration with Constrained Bayesian Optimization

Robotic Exploration using Generalized Behavioral Entropy

Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning

Maximum Entropy Reinforcement Learning with Evolution Strategies

Risk-Sensitive Reinforcement Learning with Exponential Criteria

Towards Efficient Risk-Sensitive Policy Gradient: An Iteration Complexity Analysis

Cautious Bayesian Optimization for Efficient and Scalable Policy Search

On the Global Convergence of Risk-Averse Policy Gradient Methods with Expected Conditional Risk Measures

Reinforcement Learning with Dynamic Convex Risk Measures

Risk-Sensitive Reinforcement Learning via Policy Gradient Search

Deterministic Policy Optimization by Combining Pathwise and Score Function Estimators for Discrete Action Spaces

Risk-Averse Biased Human Policies in Assistive Multi-Armed Bandit Settings

Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization

Extremum-Seeking Action Selection for Accelerating Policy Optimization

A policy gradient approach for optimization of smooth risk measures

Mitigating optimistic bias in entropic risk estimation and optimization with an application to insurance

The Exploration-Exploitation Dilemma Revisited: An Entropy Perspective

Policy Gradients with Variance Related Risk Criteria

Efficient Risk-Averse Reinforcement Learning

Epistemic Risk-Sensitive Reinforcement Learning