Abstract:Reinforcement learning (RL) theory has largely focused on proving minimax sample complexity bounds. These require strategic exploration algorithms that use relatively limited function classes for representing the policy or value function. Our goal is to explain why deep RL algorithms often perform well in practice, despite using random exploration and much more expressive function classes like neural networks. Our work arrives at an explanation by showing that many stochastic MDPs can be solved by performing only a few steps of value iteration on the random policy's Q function and then acting greedily. When this is true, we find that it is possible to separate the exploration and learning components of RL, making it much easier to analyze. We introduce a new RL algorithm, SQIRL, that iteratively learns a near-optimal policy by exploring randomly to collect rollouts and then performing a limited number of steps of fitted-Q iteration over those rollouts. Any regression algorithm that satisfies basic in-distribution generalization properties can be used in SQIRL to efficiently solve common MDPs. This can explain why deep RL works, since it is empirically established that neural networks generalize well in-distribution. Furthermore, SQIRL explains why random exploration works well in practice. We leverage SQIRL to derive instance-dependent sample complexity bounds for RL that are exponential only in an "effective horizon" of lookahead and on the complexity of the class used for function approximation. Empirically, we also find that SQIRL performance strongly correlates with PPO and DQN performance in a variety of stochastic environments, supporting that our theoretical analysis is predictive of practical performance. Our code and data are available at <a class="link-external link-https" href="https://github.com/cassidylaidlaw/effective-horizon" rel="external noopener nofollow">this https URL</a>.

Double Successive Over-Relaxation Q-Learning with an Extension to Deep Reinforcement Learning

Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning

Gradient Q(σ, Λ): A Unified Algorithm with Function Approximation for Reinforcement Learning

Gradient Q : A Unified Algorithm with Function Approximation for Reinforcement Learning

Double Speedy Q-Learning Based on Successive over Relaxation

Deep Reinforcement Learning with Double Q-Learning

Simultaneous Double Q-learning with Conservative Advantage Learning for Actor-Critic Methods

Ensemble Bootstrapping for Q-Learning

Finite-Time Analysis of Simultaneous Double Q-learning

Self-correcting Q-learning.

Constant Stepsize Q-learning: Distributional Convergence, Bias and Extrapolation

Randomized Ensembled Double Q-Learning: Learning Fast Without a Model

Strategically Conservative Q-Learning

Deep Reinforcement Learning with Sarsa and Q-Learning: A Hybrid Approach

Integrated Double Estimator Architecture for Reinforcement Learning

Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning

Two-Step Q-Learning

The Effective Horizon Explains Deep RL Performance in Stochastic Environments

Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees

SDV: Simple Double Validation Model-based Offline Reinforcement Learning

Successively Pruned Q-Learning: Using Self Q-function to Reduce the Overestimation.