Abstract:Entropy regularization is an efficient technique for encouraging exploration and preventing a premature convergence of (vanilla) policy gradient methods in reinforcement learning (RL). However, the theoretical understanding of entropy-regularized RL algorithms has been limited. In this paper, we revisit the classical entropy regularized policy gradient methods with the soft-max policy parametrization, whose convergence has so far only been established assuming access to exact gradient oracles. To go beyond this scenario, we propose the first set of (nearly) unbiased stochastic policy gradient estimators with trajectory-level entropy regularization, with one being an unbiased visitation measure-based estimator and the other one being a nearly unbiased yet more practical trajectory-based estimator. We prove that although the estimators themselves are unbounded in general due to the additional logarithmic policy rewards introduced by the entropy term, the variances are uniformly bounded. We then propose a two-phase stochastic policy gradient (PG) algorithm that uses a large batch size in the first phase to overcome the challenge of the stochastic approximation due to the non-coercive landscape, and uses a small batch size in the second phase by leveraging the curvature information around the optimal policy. We establish a global optimality convergence result and a sample complexity of $\widetilde{\mathcal{O}}(\frac{1}{\epsilon^2})$ for the proposed algorithm. Our result is the first global convergence and sample complexity results for the stochastic entropy-regularized vanilla PG method.

Stochastic Information Gradient Algorithms Under (H, Phi)-Entropy Criterion

Stochastic information gradient algorithm based on maximum entropy density estimation

Adaptive Inverse Control Under (H,Phi)-Entropy Criterion

Generalization Bounds for Stochastic Gradient Langevin Dynamics: A Unified View Via Information Leakage Analysis

Stochastic Information Gradient Algorithm with Generalized Gaussian Distribution Model.

Adaptive Inverse Control under (h,ø)-Entropy Criterion

On Mutual Information, Mmse And H-Infinity Entropy In Stochastic Linear Feedback Systems

Adaptive filtering by minimizing error's entropy power

Nonparametric inference of stochastic differential equations based on the relative entropy rate

Epsilon-Entropy and H∞ Entropy in Continuous Time Systems

Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularization

A Joint Stochastic Gradient Algorithm And Its Application To System Identification With Rbf Networks

Minimum Total Error Entropy Method for Parameter Estimation

Limitations of Information-Theoretic Generalization Bounds for Gradient Descent Methods in Stochastic Convex Optimization

Generalized Minimum Error Entropy for Adaptive Filtering

Δ-Entropy: Definition, Properties and Applications in System Identification with Quantized Data

Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Renyi's Entropy Perspective

Entropy-based convergence rates of greedy algorithms

Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning

Information Theoretic Interpretation of Error Criteria

Relations between entropy rate, entropy production and information geometry in linear stochastic systems