Abstract:Policy gradient algorithms are an important family of deep reinforcement learning techniques. Many past research endeavors focused on using the first-order policy gradient information to train policy networks. Different from these works, we conduct research in this paper driven by the believe that properly utilizing and controlling Hessian information associated with the policy gradient can noticeably improve the performance of policy gradient algorithms. One key Hessian information that attracted our attention is the Hessian trace, which gives the divergence of the policy gradient vector field in the Euclidean policy parametric space. We set the goal to generalize this Euclidean policy parametric space into a general Riemmanian manifold by introducing a metric tensor field $g_ab$ in the parametric space. This is achieved through newly developed mathematical tools, deep learning algorithms, and metric tensor deep neural networks (DNNs). Armed with these technical developments, we propose a new policy gradient algorithm that learns to minimize the absolute divergence in the Riemannian manifold as an important regularization mechanism, allowing the Riemannian manifold to smoothen its policy gradient vector field. The newly developed algorithm is experimentally studied on several benchmark reinforcement learning problems. Our experiments clearly show that the new metric tensor regularized algorithm can significantly outperform its counterpart that does not use our regularization technique. Additional experimental analysis further suggests that the trained metric tensor DNN and the corresponding metric tensor $g_{ab}$ can effectively reduce the absolute divergence towards zero in the Riemannian manifold.

Hessian Aided Policy Gradient

Sample Complexity of Policy Gradient Finding Second-Order Stationary Points

Policy Optimization with Stochastic Mirror Descent.

Stochastic Cubic-Regularized Policy Gradient Method

Efficient sample reuse in policy gradients with parameter-based exploration

Improved Sample Complexity Analysis of Natural Policy Gradient Algorithm with General Parameterization for Infinite Horizon Discounted Reward Markov Decision Processes

Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policies

Global Convergence of Natural Policy Gradient with Hessian-aided Momentum Variance Reduction

Deep Metric Tensor Regularized Policy Gradient

Elementary Analysis of Policy Gradient Methods

Policy Mirror Descent Inherently Explores Action Space

Variance Reduction based Partial Trajectory Reuse to Accelerate Policy Gradient Optimization

Variational Policy Gradient Method for Reinforcement Learning with General Utilities

Variance-Reduced Policy Gradient Approaches for Infinite Horizon Average Reward Markov Decision Processes

Identifying Policy Gradient Subspaces

A Temporal-Difference Approach to Policy Gradient Estimation

Stochastic Variance-Reduced Policy Gradient

The Reinforce Policy Gradient Algorithm Revisited

Matrix Low-Rank Approximation For Policy Gradient Methods

Efficient Wasserstein Natural Gradients for Reinforcement Learning