Abstract:Distributional reinforcement learning (DRL) enhances the understanding of the effects of the randomness in the environment by letting agents learn the distribution of a random return, rather than its expected value as in standard reinforcement learning. Meanwhile, a challenge in DRL is that the policy evaluation typically relies on the representation of the return distribution, which needs to be carefully designed. In this paper, we address this challenge for the special class of DRL problems that rely on a discounted linear quadratic regulator (LQR), which we call \emph{distributional LQR}. Specifically, we provide a closed-form expression for the distribution of the random return, which is applicable for all types of exogenous disturbance as long as it is independent and identically distributed (i.i.d.). We show that the variance of the random return is bounded if the fourth moment of the exogenous disturbance is bounded. Furthermore, we investigate the sensitivity of the return distribution to model perturbations. While the proposed exact return distribution consists of infinitely many random variables, we show that this distribution can be well approximated by a finite number of random variables. The associated approximation error can be analytically bounded under mild assumptions. When the model is unknown, we propose a model-free approach for estimating the return distribution, supported by sample complexity guarantees. Finally, we extend our approach to partially observable linear systems. Numerical experiments are provided to illustrate the theoretical results.

Distributional reinforcement learning with epistemic and aleatoric uncertainty estimation

Exploration Via Distributional Reinforcement Learning with Epistemic and Aleatoric Uncertainty Estimation*

Distributional Reinforcement Learning for Efficient Exploration

Distributional Reinforcement Learning With Quantile Regression

Distributional Soft Actor Critic for Risk Sensitive Learning

Fully Parameterized Quantile Function for Distributional Reinforcement Learning.

Uncertainty-aware Distributional Offline Reinforcement Learning

Variance Control for Distributional Reinforcement Learning.

Variance Control for Distributional Reinforcement Learning

Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning

Distributional Reinforcement Learning for Multi-Dimensional Reward Functions

Value-Distributional Model-Based Reinforcement Learning

Towards Understanding Distributional Reinforcement Learning: Regularization, Optimization, Acceleration and Sinkhorn Algorithm

Echoes of Socratic Doubt: Embracing Uncertainty in Calibrated Evidential Reinforcement Learning

A unified uncertainty-aware exploration: Combining epistemic and aleatory uncertainty

Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors

Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model

Distributionally Robust Constrained Reinforcement Learning under Strong Duality

Interpreting Distributional Reinforcement Learning: A Regularization Perspective

Enabling risk-aware Reinforcement Learning for medical interventions through uncertainty decomposition

Policy Evaluation in Distributional LQR (Extended Version)