Abstract:Distributional reinforcement learning (DRL) has achieved empirical success in various domains. One of the core tasks in the field of DRL is distributional policy evaluation, which involves estimating the return distribution $\eta^\pi$ for a given policy $\pi$. The distributional temporal difference (TD) algorithm has been accordingly proposed, which is an extension of the temporal difference algorithm in the classic RL literature. In the tabular case, \citet{rowland2018analysis} and \citet{rowland2023analysis} proved the asymptotic convergence of two instances of distributional TD, namely categorical temporal difference algorithm (CTD) and quantile temporal difference algorithm (QTD), respectively. In this paper, we go a step further and analyze the finite-sample performance of distributional TD. To facilitate theoretical analysis, we propose a non-parametric distributional TD algorithm (NTD). For a $\gamma$-discounted infinite-horizon tabular Markov decision process, we show that for NTD we need $\tilde{O}\left(\frac{1}{\varepsilon^{2p}(1-\gamma)^{2p+1}}\right)$ iterations to achieve an $\varepsilon$-optimal estimator with high probability, when the estimation error is measured by the $p$-Wasserstein distance. This sample complexity bound is minimax optimal (up to logarithmic factors) in the case of the $1$-Wasserstein distance. To achieve this, we establish a novel Freedman's inequality in Hilbert spaces, which would be of independent interest. In addition, we revisit CTD, showing that the same non-asymptotic convergence bounds hold for CTD in the case of the $p$-Wasserstein distance.

CLT, MDP and LDP for Range-Renewals of I.I.D.Samplings from an Infinite Discrete Distribution

LIL for the Length of the Longest Increasing Subsequences

Range-Renewal Structure of I.I.D. Samplings

A Supplement to the Complete Convergence

Precise Rates in the Law of the Iterated Logarithm under Dependence Assumptions

On Increments of Sums of Independent Non-Identically Distributed Random-Variables

Range-Renewal Processes: SLLN, Power Law and Beyonds

Risk Bounds for Infinitely Divisible Distribution

Range-Renewal Structure of Transient Simple Random Walk

The Limit Points of the Strong Law of Large Numbers under the Sub-Linear Expectations

A Limit Result for Self-Normalized Random Sums

Precise Rates in the Generalized Law of the Iterated Logarithm

Sufficient and Necessary Conditions for Limit Laws on Lag Sums of Iidrvs

Central Limit Theorem for Rényi Divergence of Infinite Order

Range-Renewal Processes: SLLNs and Power Laws

Limiting Behaviour of Poisson-Dirichlet and Generalised Poisson-Dirichlet Distributions

A Proof of Basic Limit Theorem of Renewal Theory

Extension of Limit Theory with Deleting Items Partial Sum of Random Variable Sequence

Reliable Force Aggregation Using a Refined Evidence Specification from Dempster-Shafer Clustering

Near Minimax-Optimal Distributional Temporal Difference Algorithms and The Freedman Inequality in Hilbert Spaces

Central limit theorem for the random variables associated with the IDS of the Anderson model on lattice