Abstract:To facilitate efficient learning, policy gradient approaches to deep reinforcement learning (RL) are typically paired with variance reduction measures and strategies for making large but safe policy changes based on a batch of experiences. Natural policy gradient methods, including Trust Region Policy Optimization (TRPO), seek to produce monotonic improvement through bounded changes in policy outputs. Proximal Policy Optimization (PPO) is a commonly used, first-order algorithm that instead uses loss clipping to take multiple safe optimization steps per batch of data, replacing the bound on the single step of TRPO with regularization on multiple steps. In this work, we find that the performance of PPO, when applied to continuous action spaces, may be consistently improved through a simple change in objective. Instead of the importance sampling objective of PPO, we instead recommend a basic policy gradient, clipped in an equivalent fashion. While both objectives produce biased gradient estimates with respect to the RL objective, they also both display significantly reduced variance compared to the unbiased off-policy policy gradient. Additionally, we show that (1) the clipped-objective policy gradient (COPG) objective is on average "pessimistic" compared to both the PPO objective and (2) this pessimism promotes enhanced exploration. As a result, we empirically observe that COPG produces improved learning compared to PPO in single-task, constrained, and multi-task learning, without adding significant computational cost or complexity. Compared to TRPO, the COPG approach is seen to offer comparable or superior performance, while retaining the simplicity of a first-order method.

Technical Report: The Policy Graph Improvement Algorithm

Policy Graph Pruning And Optimization In Monte Carlo Value Iteration For Continuous-State Pomdps

Generalised Policy Improvement with Geometric Policy Composition

The policy graph decomposition of multistage stochastic programming problems

Anytime Point-Based Approximations for Large POMDPs

Processing Network Controls via Deep Reinforcement Learning

Explaining the Behaviour of Reinforcement Learning Agents in a Multi-Agent Cooperative Environment Using Policy Graphs

The Policy Gradient Estimation for Continuous-Time Partially Observable Markovian Decision Processes

Point-Based POMDP Algorithms: Improved Analysis and Implementation

Policy Search for the Optimal Control of Markov Decision Processes: A Novel Particle-Based Iterative Scheme

Policy Optimization with Model-based Explorations

An Analytical Update Rule for General Policy Optimization

Simulation Optimization of Actions of Robot Based on POMDP Model

Policy iteration for discrete-time systems with discounted costs: stability and near-optimality guarantees

Monte Carlo Sampling Methods for Approximating Interactive POMDPs

Online algorithms for POMDPs with continuous state, action, and observation spaces

Beyond the Boundaries of Proximal Policy Optimization

Entropy adjustment by interpolation for exploration in Proximal Policy Optimization (PPO)

A Theoretical Analysis of Optimistic Proximal Policy Optimization in Linear Markov Decision Processes

On the Performance Bounds of some Policy Search Dynamic Programming Algorithms

Clipped-Objective Policy Gradients for Pessimistic Policy Optimization