Abstract:While numerous works have focused on devising efficient algorithms for reinforcement learning (RL) with uniformly bounded rewards, it remains an open question whether sample or time-efficient algorithms for RL with large state-action space exist when the rewards are \emph{heavy-tailed}, i.e., with only finite $(1+\epsilon)$-th moments for some $\epsilon\in(0,1]$. In this work, we address the challenge of such rewards in RL with linear function approximation. We first design an algorithm, \textsc{Heavy-OFUL}, for heavy-tailed linear bandits, achieving an \emph{instance-dependent} $T$-round regret of $\tilde{O}\big(d T^{\frac{1-\epsilon}{2(1+\epsilon)}} \sqrt{\sum_{t=1}^T \nu_t^2} + d T^{\frac{1-\epsilon}{2(1+\epsilon)}}\big)$, the \emph{first} of this kind. Here, $d$ is the feature dimension, and $\nu_t^{1+\epsilon}$ is the $(1+\epsilon)$-th central moment of the reward at the $t$-th round. We further show the above bound is minimax optimal when applied to the worst-case instances in stochastic and deterministic linear bandits. We then extend this algorithm to the RL settings with linear function approximation. Our algorithm, termed as \textsc{Heavy-LSVI-UCB}, achieves the \emph{first} computationally efficient \emph{instance-dependent} $K$-episode regret of $\tilde{O}(d \sqrt{H \mathcal{U}^*} K^\frac{1}{1+\epsilon} + d \sqrt{H \mathcal{V}^* K})$. Here, $H$ is length of the episode, and $\mathcal{U}^*, \mathcal{V}^*$ are instance-dependent quantities scaling with the central moment of reward and value functions, respectively. We also provide a matching minimax lower bound $\Omega(d H K^{\frac{1}{1+\epsilon}} + d \sqrt{H^3 K})$ to demonstrate the optimality of our algorithm in the worst case. Our result is achieved via a novel robust self-normalized concentration inequality that may be of independent interest in handling heavy-tailed noise in general online regression problems.

VOL: Towards Optimal Regret in Model-free RL with Nonlinear Function Approximation

Tackling Heavy-Tailed Rewards in Reinforcement Learning with Function Approximation: Minimax Optimal and Instance-Dependent Regret Bounds

Nonstationary Reinforcement Learning with Linear Function Approximation

Nearly Minimax Optimal Reinforcement Learning with Linear Function Approximation.

Horizon-Free and Instance-Dependent Regret Bounds for Reinforcement Learning with General Function Approximation

Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learning

Horizon-Free and Variance-Dependent Reinforcement Learning for Latent Markov Decision Processes

Provably Efficient Reinforcement Learning with Linear Function Approximation

Kernel-Based Function Approximation for Average Reward Reinforcement Learning: An Optimist No-Regret Algorithm

$\Sqrt{n}$-Regret for Learning in Markov Decision Processes with Function Approximation and Low Bellman Rank

Sublinear Regret for a Class of Continuous-Time Linear--Quadratic Reinforcement Learning Problems

Near-Optimal Regret in Linear MDPs with Aggregate Bandit Feedback

A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximation

Optimism in Reinforcement Learning with Generalized Linear Function Approximation.

Non-stationary Reinforcement Learning under General Function Approximation

Provably Efficient Infinite-Horizon Average-Reward Reinforcement Learning with Linear Function Approximation

Posterior Sampling with Delayed Feedback for Reinforcement Learning with Linear Function Approximation

√N-Regret for Learning in Markov Decision Processes with Function Approximation and Low Bellman Rank.

Near-Optimal Representation Learning For Linear Bandits And Linear Rl

Regret Minimization For Reinforcement Learning By Evaluating The Optimal Bias Function

Generalized linear function approximation