Abstract:To facilitate efficient learning, policy gradient approaches to deep reinforcement learning (RL) are typically paired with variance reduction measures and strategies for making large but safe policy changes based on a batch of experiences. Natural policy gradient methods, including Trust Region Policy Optimization (TRPO), seek to produce monotonic improvement through bounded changes in policy outputs. Proximal Policy Optimization (PPO) is a commonly used, first-order algorithm that instead uses loss clipping to take multiple safe optimization steps per batch of data, replacing the bound on the single step of TRPO with regularization on multiple steps. In this work, we find that the performance of PPO, when applied to continuous action spaces, may be consistently improved through a simple change in objective. Instead of the importance sampling objective of PPO, we instead recommend a basic policy gradient, clipped in an equivalent fashion. While both objectives produce biased gradient estimates with respect to the RL objective, they also both display significantly reduced variance compared to the unbiased off-policy policy gradient. Additionally, we show that (1) the clipped-objective policy gradient (COPG) objective is on average "pessimistic" compared to both the PPO objective and (2) this pessimism promotes enhanced exploration. As a result, we empirically observe that COPG produces improved learning compared to PPO in single-task, constrained, and multi-task learning, without adding significant computational cost or complexity. Compared to TRPO, the COPG approach is seen to offer comparable or superior performance, while retaining the simplicity of a first-order method.

Towards Combining On-Off-Policy Methods for Real-World Applications

An Off-Policy Trust Region Policy Optimization Method with Monotonic Improvement Guarantee for Deep Reinforcement Learning

Training Efficient Controllers via Analytic Policy Gradient

OMPO: A Unified Framework for RL under Policy and Dynamics Shifts

Clipped-Objective Policy Gradients for Pessimistic Policy Optimization

Unified Policy Optimization for Continuous-action Reinforcement Learning in Non-stationary Tasks and Games

Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning

Adaptive Gain Scheduling using Reinforcement Learning for Quadcopter Control

Proximal Policy Optimization Algorithms

Relative Policy-Transition Optimization for Fast Policy Transfer

Residual Policy Learning for Perceptive Quadruped Control Using Differentiable Simulation

Augmented Proximal Policy Optimization for Safe Reinforcement Learning

Accelerating Proximal Policy Optimization Learning Using Task Prediction for Solving Environments with Delayed Rewards

Policy Optimization with Model-based Explorations

Authentic Boundary Proximal Policy Optimization

Adversarial Constrained Policy Optimization: Improving Constrained Reinforcement Learning by Adapting Budgets

Beyond the Boundaries of Proximal Policy Optimization

Mean policy-based proximal policy optimization for maneuvering decision in multi-UAV air combat

An off-policy multi-agent stochastic policy gradient algorithm for cooperative continuous control

OGMP: Oracle Guided Multi-mode Policies for Agile and Versatile Robot Control

POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decomposition