Abstract:Deep Deterministic Policy Gradient (DDPG) algorithm is one of the most well-known reinforcement learning methods. However, this method is inefficient and unstable in practical applications. On the other hand, the bias and variance of the Q estimation in the target function are sometimes difficult to control. This paper proposes a Regularly Updated Deterministic (RUD) policy gradient algorithm for these problems. This paper theoretically proves that the learning procedure with RUD can make better use of new data in replay buffer than the traditional procedure. In addition, the low variance of the Q value in RUD is more suitable for the current Clipped Double Q-learning strategy. This paper has designed a comparison experiment against previous methods, an ablation experiment with the original DDPG, and other analytical experiments in Mujoco environments. The experimental results demonstrate the effectiveness and superiority of RUD.

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly focus on the inefficiency and instability of the Deep Deterministic Policy Gradient (DDPG) algorithm in practical applications, as well as the difficulty in controlling the bias and variance of Q - value estimation in the objective function. To address these problems, the paper proposes the Regularly Updated Deterministic (RUD) policy gradient algorithm. Specifically, the paper focuses on the following points: 1. **Improving data utilization**: Traditional offline reinforcement learning methods have deficiencies in utilizing new experiences, resulting in low learning efficiency. RUD alleviates this problem by allowing the exploration and learning processes to be alternately concentrated, enabling new experiences to be utilized more effectively. 2. **Reducing the bias of Q - value estimation**: The paper analyzes the downward bias and its magnitude existing in the Clipped Double Q - learning method and proves that the RUD process can mitigate this bias, thereby improving the accuracy of target value estimation. 3. **Balancing exploration and exploitation**: RUD balances exploration efficiency and data utilization by setting the parameter F. A larger F value can improve data utilization but reduce exploration efficiency, while a smaller F value is the opposite. The paper verifies the influence of different F values on the performance of the algorithm through experiments and finds a relatively ideal balance point. Through these improvements, the experimental results of RUD in multiple Mujoco environments show that it not only exhibits higher stability and average performance in complex tasks, but also outperforms existing methods in terms of data utilization and accuracy of target value estimation.

Regularly Updated Deterministic Policy Gradient Algorithm

Multi-Agent Deep Deterministic Policy Gradient Algorithm Based on Classification Experience Replay

Dueling Network Architecture for Multi-Agent Deep Deterministic Policy Gradient

Deterministic Value-Policy Gradients

Stochastic Cubic-Regularized Policy Gradient Method

Deep Deterministic Policy Gradient With Clustered Prioritized Sampling

Asynchronous Episodic Deep Deterministic Policy Gradient: Toward Continuous Control in Computationally Complex Environments

Deep Deterministic Policy Gradient with Episode Experience Replay

Boltzmann Exploration for Deterministic Policy Optimization.

Network Architecture for Optimizing Deep Deterministic Policy Gradient Algorithms

Edge Delayed Deep Deterministic Policy Gradient: efficient continuous control for edge scenarios

Swap Softmax Twin Delayed Deep Deterministic Policy Gradient

Exponential Moving Averaged Q-Network for DDPG

An Improved Prioritized DDPG Based on Fractional-Order Learning Scheme

Revisiting Estimation Bias in Policy Gradients for Deep Reinforcement Learning

Alternated Greedy-Step Deterministic Policy Gradient

Heuristic Gait Learning of Quadruped Robot Based on Deep Deterministic Policy Gradient Algorithm

Deterministic Policy Gradients with General State Transitions

Enhanced Deep Deterministic Policy Gradient Algorithm

Self-Driving Via Improved DDPG Algorithm

Policy ensemble gradient for continuous control problems in deep reinforcement learning