Abstract:Reinforcement learning (RL) theory has largely focused on proving minimax sample complexity bounds. These require strategic exploration algorithms that use relatively limited function classes for representing the policy or value function. Our goal is to explain why deep RL algorithms often perform well in practice, despite using random exploration and much more expressive function classes like neural networks. Our work arrives at an explanation by showing that many stochastic MDPs can be solved by performing only a few steps of value iteration on the random policy's Q function and then acting greedily. When this is true, we find that it is possible to separate the exploration and learning components of RL, making it much easier to analyze. We introduce a new RL algorithm, SQIRL, that iteratively learns a near-optimal policy by exploring randomly to collect rollouts and then performing a limited number of steps of fitted-Q iteration over those rollouts. Any regression algorithm that satisfies basic in-distribution generalization properties can be used in SQIRL to efficiently solve common MDPs. This can explain why deep RL works, since it is empirically established that neural networks generalize well in-distribution. Furthermore, SQIRL explains why random exploration works well in practice. We leverage SQIRL to derive instance-dependent sample complexity bounds for RL that are exponential only in an "effective horizon" of lookahead and on the complexity of the class used for function approximation. Empirically, we also find that SQIRL performance strongly correlates with PPO and DQN performance in a variety of stochastic environments, supporting that our theoretical analysis is predictive of practical performance. Our code and data are available at <a class="link-external link-https" href="https://github.com/cassidylaidlaw/effective-horizon" rel="external noopener nofollow">this https URL</a>.

Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear

Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings

Towards Safe Reinforcement Learning Via Constraining Conditional Value-at-Risk

Shielding Atari Games with Bounded Prescience

Adaptive Deep Reinforcement Learning for Non-Stationary Environments

Success-Rate Targeted Reinforcement Learning by Disorientation Penalty

Self Punishment and Reward Backfill for Deep Q-Learning

Safe Reinforcement Learning via Hierarchical Adaptive Chance-Constraint Safeguards

Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation

Random curiosity-driven exploration in deep reinforcement learning

Fear-Neuro-Inspired Reinforcement Learning for Safe Autonomous Driving

Learning to be Safe: Deep RL with a Safety Critic

Survival-Oriented Reinforcement Learning Model: an Effcient and Robust Deep Reinforcement Learning Algorithm for Autonomous Driving Problem.

The Effective Horizon Explains Deep RL Performance in Stochastic Environments

Dissecting Deep RL with High Update Ratios: Combatting Value Divergence

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

Distance-rank Aware Sequential Reward Learning for Inverse Reinforcement Learning with Sub-optimal Demonstrations

Does DQN Learn?

An Information-Theoretic Optimality Principle for Deep Reinforcement Learning

Learning Sparse Representations Incrementally in Deep Reinforcement Learning

Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning