Abstract:This paper provides a theoretical study of deep neural function approximation in reinforcement learning (RL) with the $\epsilon$-greedy exploration under the online setting. This problem setting is motivated by the successful deep Q-networks (DQN) framework that falls in this regime. In this work, we provide an initial attempt on theoretical understanding deep RL from the perspective of function class and neural networks architectures (e.g., width and depth) beyond the ``linear'' regime. To be specific, we focus on the value based algorithm with the $\epsilon$-greedy exploration via deep (and two-layer) neural networks endowed by Besov (and Barron) function spaces, respectively, which aims at approximating an $\alpha$-smooth Q-function in a $d$-dimensional feature space. We prove that, with $T$ episodes, scaling the width $m = \widetilde{\mathcal{O}}(T^{\frac{d}{2\alpha + d}})$ and the depth $L=\mathcal{O}(\log T)$ of the neural network for deep RL is sufficient for learning with sublinear regret in Besov spaces. Moreover, for a two layer neural network endowed by the Barron space, scaling the width $\Omega(\sqrt{T})$ is sufficient. To achieve this, the key issue in our analysis is how to estimate the temporal difference error under deep neural function approximation as the $\epsilon$-greedy exploration is not enough to ensure ``optimism''. Our analysis reformulates the temporal difference error in an $L^2(\mathrm{d}\mu)$-integrable space over a certain averaged measure $\mu$, and transforms it to a generalization problem under the non-iid setting. This might have its own interest in RL theory for better understanding $\epsilon$-greedy exploration in deep RL.

Provably Efficient Reinforcement Learning with General Value Function Approximation.

Provably Efficient Reinforcement Learning with Linear Function Approximation

Reinforcement Learning with General Value Function Approximation: Provably Efficient Approach via Bounded Eluder Dimension

Provably Efficient Reinforcement Learning Via Surprise Bound

On Reward-Free Reinforcement Learning with Linear Function Approximation

Tractable and Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation

Randomized Exploration for Reinforcement Learning with General Value Function Approximation

Gradient Q : A Unified Algorithm with Function Approximation for Reinforcement Learning

Online Sub-Sampling for Reinforcement Learning with General Function Approximation

Greedy Feature Replacement for Online Value Function Approximation

Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation.

Provably Efficient Cooperative Multi-Agent Reinforcement Learning with Function Approximation

A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximation

Generalized linear function approximation

The Optimal Approximation Factors in Misspecified Off-Policy Value Function Estimation

Foresee then Evaluate: Decomposing Value Estimation with Latent Future Prediction.

Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learning

Reward-Free Model-Based Reinforcement Learning with Linear Function Approximation

Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedback

Understanding Deep Neural Function Approximation in Reinforcement Learning via $ε$-Greedy Exploration

Avoiding Confusion between Predictors and Inhibitors in Value Function Approximation