Abstract:Distributional reinforcement learning (RL) is a class of state-of-the-art algorithms that estimate the whole distribution of the total return rather than only its expectation. Despite the remarkable performance of distributional RL, a theoretical understanding of its advantages over expectation-based RL remains elusive. In this paper, we interpret distributional RL as entropy-regularized maximum likelihood estimation in the neural Z-fitted iteration framework, and establish the connection of the resulting risk-aware regularization with maximum entropy RL. In addition, We shed light on the stability-promoting distributional loss with desirable smoothness properties in distributional RL, which can yield stable optimization and guaranteed generalization. We also analyze the acceleration behavior while optimizing distributional RL algorithms and show that an appropriate approximation to the true target distribution can speed up the convergence. From the perspective of representation, we find that distributional RL encourages state representation from the same action class classified by the policy in tighter clusters. Finally, we propose a class of Sinkhorn distributional RL algorithm that interpolates between the Wasserstein distance and maximum mean discrepancy (MMD). Experiments on a suite of Atari games reveal the competitive performance of our algorithm relative to existing state-of-the-art distributional RL algorithms.

Distributional Reinforcement Learning for Efficient Exploration

Exploration Via Distributional Reinforcement Learning with Epistemic and Aleatoric Uncertainty Estimation*

Distributional reinforcement learning with epistemic and aleatoric uncertainty estimation

Non-crossing quantile regression for deep reinforcement learning

Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning

Exploration and Anti-Exploration with Distributional Random Network Distillation

Distributional Reinforcement Learning With Quantile Regression

Exploration by Distributional Reinforcement Learning

Distributional Reinforcement Learning for Multi-Dimensional Reward Functions

Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learning

Fully Parameterized Quantile Function for Distributional Reinforcement Learning.

Efficient and Scalable Exploration Via Estimation-Error

Distributional Soft Actor Critic for Risk Sensitive Learning

Variance Control for Distributional Reinforcement Learning.

Pseudo Value Network Distillation for High-Performance Exploration

Information-Directed Exploration for Deep Reinforcement Learning

Towards Understanding Distributional Reinforcement Learning: Regularization, Optimization, Acceleration and Sinkhorn Algorithm

Single-Trajectory Distributionally Robust Reinforcement Learning

Variance Control for Distributional Reinforcement Learning

DSAC: Distributional Soft Actor Critic for Risk-Sensitive Reinforcement Learning

Reward Uncertainty for Exploration in Preference-based Reinforcement Learning