Abstract:We revisit the estimation bias in policy gradients for the discounted episodic Markov decision process (MDP) from Deep Reinforcement Learning (DRL) perspective. The objective is formulated theoretically as the expected returns discounted over the time horizon. One of the major policy gradient biases is the state distribution shift: the state distribution used to estimate the gradients differs from the theoretical formulation in that it does not take into account the discount factor. Existing discussion of the influence of this bias was limited to the tabular and softmax cases in the literature. Therefore, in this paper, we extend it to the DRL setting where the policy is parameterized and demonstrate how this bias can lead to suboptimal policies theoretically. We then discuss why the empirically inaccurate implementations with shifted state distribution can still be effective. We show that, despite such state distribution shift, the policy gradient estimation bias can be reduced in the following three ways: 1) a small learning rate; 2) an adaptive-learning-rate-based optimizer; and 3) KL regularization. Specifically, we show that a smaller learning rate, or, an adaptive learning rate, such as that used by Adam and RSMProp optimizers, makes the policy optimization robust to the bias. We further draw connections between optimizers and the optimization regularization to show that both the KL and the reverse KL regularization can significantly rectify this bias. Moreover, we provide extensive experiments on continuous control tasks to support our analysis. Our paper sheds light on how successful PG algorithms optimize policies in the DRL setting, and contributes insights into the practical issues in DRL.

Kernel-Based Decentralized Policy Evaluation for Reinforcement Learning

Finite-Sample Analysis of Decentralized Temporal-Difference Learning with Linear Function Approximation

Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policies

Fully asynchronous policy evaluation in distributed reinforcement learning over networks

Reinforcement Leaning for Infinite-Dimensional Systems

General Policy Evaluation and Improvement by Learning to Identify Few But Crucial States

Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning

Kernel dynamic policy programming: Applicable reinforcement learning to robot systems with high dimensional states

Kalman meets Bellman: Improving Policy Evaluation through Value Tracking

Manifold Regularization for Kernelized LSTD

An L2 Analysis of Reinforcement Learning in High Dimensions with Kernel and Neural Network Approximation

Kernel-Based Function Approximation for Average Reward Reinforcement Learning: An Optimist No-Regret Algorithm

Decentralized Natural Policy Gradient with Variance Reduction for Collaborative Multi-Agent Reinforcement Learning

Policy Optimization over General State and Action Spaces

GOV-REK: Governed Reward Engineering Kernels for Designing Robust Multi-Agent Reinforcement Learning Systems

Decentralized Multi-Agent Reinforcement Learning: An Off-Policy Method

Revisiting Estimation Bias in Policy Gradients for Deep Reinforcement Learning

A Decentralized Policy Gradient Approach to Multi-task Reinforcement Learning

On The Convergence Of Policy Iteration-Based Reinforcement Learning With Monte Carlo Policy Evaluation

Policy evaluation for reinforcement learning over asynchronous multi-agent networks