Behind the Myth of Exploration in Policy Gradients

Adrien Bolland,Gaspard Lambrechts,Damien Ernst

2024-02-01

Abstract:Policy-gradient algorithms are effective reinforcement learning methods for solving control problems with continuous state and action spaces. To compute near-optimal policies, it is essential in practice to include exploration terms in the learning objective. Although the effectiveness of these terms is usually justified by an intrinsic need to explore environments, we propose a novel analysis and distinguish two different implications of these techniques. First, they make it possible to smooth the learning objective and to eliminate local optima while preserving the global maximum. Second, they modify the gradient estimates, increasing the probability that the stochastic parameter update eventually provides an optimal policy. In light of these effects, we discuss and illustrate empirically exploration strategies based on entropy bonuses, highlighting their limitations and opening avenues for future works in the design and analysis of such strategies.

Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is about the understanding of the effectiveness and mechanism of the exploration mechanism in the policy gradient algorithm. Specifically, through analysis and experiments, the paper explores two main influences of the exploration mechanism in the policy gradient method:

1. **Smoothing the learning objective function**: The exploration mechanism can help smooth the learning objective function, eliminate local optima, and at the same time retain the global maximum. This means that exploration not only helps the algorithm avoid getting trapped in local optima, but also maintains the ability to find the global optimum.

2. **Modifying the gradient estimate**: The exploration mechanism modifies the gradient estimate by increasing the probability of randomly parameter - updated final optimal policies. This shows that through appropriate exploration strategies, the efficiency of the policy gradient algorithm in finding the optimal policy can be improved.

In addition, the paper also discusses the exploration strategies based on entropy rewards, points out the limitations of these strategies, and provides directions for the future design and analysis of such strategies. The main contribution of the paper lies in distinguishing different influences of the exploration mechanism and conducting a strict analysis of these influences from the perspective of optimization theory, thus providing a new perspective for understanding and improving the exploration mechanism in the policy gradient algorithm.