Abstract:We revisit the estimation bias in policy gradients for the discounted episodic Markov decision process (MDP) from Deep Reinforcement Learning (DRL) perspective. The objective is formulated theoretically as the expected returns discounted over the time horizon. One of the major policy gradient biases is the state distribution shift: the state distribution used to estimate the gradients differs from the theoretical formulation in that it does not take into account the discount factor. Existing discussion of the influence of this bias was limited to the tabular and softmax cases in the literature. Therefore, in this paper, we extend it to the DRL setting where the policy is parameterized and demonstrate how this bias can lead to suboptimal policies theoretically. We then discuss why the empirically inaccurate implementations with shifted state distribution can still be effective. We show that, despite such state distribution shift, the policy gradient estimation bias can be reduced in the following three ways: 1) a small learning rate; 2) an adaptive-learning-rate-based optimizer; and 3) KL regularization. Specifically, we show that a smaller learning rate, or, an adaptive learning rate, such as that used by Adam and RSMProp optimizers, makes the policy optimization robust to the bias. We further draw connections between optimizers and the optimization regularization to show that both the KL and the reverse KL regularization can significantly rectify this bias. Moreover, we provide extensive experiments on continuous control tasks to support our analysis. Our paper sheds light on how successful PG algorithms optimize policies in the DRL setting, and contributes insights into the practical issues in DRL.

Infinite-Horizon Policy-Gradient Estimation with Variable Discount Factor for Markov Decision Process

Infinite-horizon gradient estimation for semi-Markov decision processes

Safe Reinforcement Learning Using Finite-Horizon Gradient-based Estimation

Variance-Reduced Policy Gradient Approaches for Infinite Horizon Average Reward Markov Decision Processes

A policy gradient approach for Finite Horizon Constrained Markov Decision Processes

The Policy Gradient Estimation for Continuous-Time Partially Observable Markovian Decision Processes

Revisiting Estimation Bias in Policy Gradients for Deep Reinforcement Learning

On the Use of Non-Stationary Policies for Infinite-Horizon Discounted Markov Decision Processes

On the Use of Non-Stationary Policies for Stationary Infinite-Horizon Markov Decision Processes

The policy gradient estimation of continuous-time hidden Markov decision processes

Theoretical Guarantees of Fictitious Discount Algorithms for Episodic Reinforcement Learning and Global Convergence of Policy Gradient Methods

Structure Matters: Dynamic Policy Gradient

Performance Optimization of Semi-Markov Decision Processes with Discounted-cost Criteria.

Markov Decision Processes with Time-Varying Geometric Discounting

Landscape of Policy Optimization for Finite Horizon MDPs with General State and Action

Two-Timescale Simulation-based Algorithm for Markov Decision Process Based on Performance Potentials

Natural Policy Gradient Primal-Dual Method for Constrained Markov Decision Processes

Variance Reduction based Partial Trajectory Reuse to Accelerate Policy Gradient Optimization

A Temporal-Difference Approach to Policy Gradient Estimation

On-Line Policy Gradient Estimation with Multi-Step Sampling.

Deterministic Policy Gradients with General State Transitions