Abstract:Reinforcement learning (RL) theory has largely focused on proving minimax sample complexity bounds. These require strategic exploration algorithms that use relatively limited function classes for representing the policy or value function. Our goal is to explain why deep RL algorithms often perform well in practice, despite using random exploration and much more expressive function classes like neural networks. Our work arrives at an explanation by showing that many stochastic MDPs can be solved by performing only a few steps of value iteration on the random policy's Q function and then acting greedily. When this is true, we find that it is possible to separate the exploration and learning components of RL, making it much easier to analyze. We introduce a new RL algorithm, SQIRL, that iteratively learns a near-optimal policy by exploring randomly to collect rollouts and then performing a limited number of steps of fitted-Q iteration over those rollouts. Any regression algorithm that satisfies basic in-distribution generalization properties can be used in SQIRL to efficiently solve common MDPs. This can explain why deep RL works, since it is empirically established that neural networks generalize well in-distribution. Furthermore, SQIRL explains why random exploration works well in practice. We leverage SQIRL to derive instance-dependent sample complexity bounds for RL that are exponential only in an "effective horizon" of lookahead and on the complexity of the class used for function approximation. Empirically, we also find that SQIRL performance strongly correlates with PPO and DQN performance in a variety of stochastic environments, supporting that our theoretical analysis is predictive of practical performance. Our code and data are available at <a class="link-external link-https" href="https://github.com/cassidylaidlaw/effective-horizon" rel="external noopener nofollow">this https URL</a>.
What problem does this paper attempt to address?
The problem that this paper attempts to solve lies in explaining why, in practice, deep reinforcement learning (Deep RL) algorithms work well, although they use stochastic exploration strategies and complex function approximators (such as neural networks), which is inconsistent with theoretical expectations. Specifically, the paper focuses on two main gaps:
1. **Gap in exploration strategies**: Theoretically, reinforcement learning (RL) emphasizes strategic exploration, while in practice, deep RL algorithms often adopt stochastic exploration strategies, such as the ε - greedy strategy. This difference is difficult to reconcile because theoretically, it is predicted that stochastic exploration requires exponential sample complexity in the worst - case scenario.
2. **Gap in function approximators**: RL theory has difficulties in dealing with complex function approximators (such as neural networks used in deep learning). Algorithms based on Upper Confidence Bound (UCB) can only be effective in highly structured environments where the value function and the policy can be represented using simple function classes.
To bridge these two gaps, the paper introduces the concept of "Effective Horizon" and proposes a new RL algorithm - SQIRL (Shallow Q - Iteration via Reinforcement Learning). Through these methods, the paper attempts to explain why stochastic exploration can work effectively in practice and why deep RL can still be successful when using complex neural networks as function approximators.
### Main contributions
- **Effective Horizon**: The paper defines "Effective Horizon", which is a key parameter for measuring whether the Q - function greedy action of a stochastic policy can reach the optimal solution after a finite number of value iterations in a stochastic environment. If the effective horizon of an environment is low, the environment can be solved through a limited number of exploration and learning steps.
- **SQIRL algorithm**: The paper proposes the SQIRL algorithm. This algorithm collects data through stochastic exploration, then uses regression and Fitted Q - Iteration (FQI) to estimate the Q - function of the stochastic policy and performs a finite number of value iterations. In this way, the exploration and learning components can be separated, making the analysis easier.
- **Theoretical and empirical analysis**: The paper provides a theoretical analysis of the sample complexity of SQIRL and experimentally proves that the performance of SQIRL in various stochastic environments is comparable to that of deep RL algorithms such as PPO and DQN.
### Experimental results
- **Performance comparison**: In more than 150 stochastic environments, the performance of SQIRL is similar to that of PPO and DQN. In environments where both PPO and DQN can converge to the optimal policy, SQIRL can also find the optimal policy 85% of the time; while in environments where both PPO and DQN fail, SQIRL never succeeds either.
- **Verification of effective horizon**: The experimental results show that the effective horizon of most environments is low, which explains why deep RL algorithms perform well in these environments.
In conclusion, through introducing the "Effective Horizon" and the SQIRL algorithm, this paper provides a theoretical framework to explain why deep RL can be successful in practice, especially when using stochastic exploration and complex function approximators.