Abstract:The experience replay mechanism allows agents to use the experiences multiple times. In prior works, the sampling probability of the transitions was adjusted according to their importance. Reassigning sampling probabilities for every transition in the replay buffer after each iteration is highly inefficient. Therefore, experience replay prioritization algorithms recalculate the significance of a transition when the corresponding transition is sampled to gain computational efficiency. However, the importance level of the transitions changes dynamically as the policy and the value function of the agent are updated. In addition, experience replay stores the transitions are generated by the previous policies of the agent that may significantly deviate from the most recent policy of the agent. Higher deviation from the most recent policy of the agent leads to more off-policy updates, which is detrimental for the agent. In this paper, we develop a novel algorithm, Batch Prioritizing Experience Replay via KL Divergence (KLPER), which prioritizes batch of transitions rather than directly prioritizing each transition. Moreover, to reduce the off-policyness of the updates, our algorithm selects one batch among a certain number of batches and forces the agent to learn through the batch that is most likely generated by the most recent policy of the agent. We combine our algorithm with Deep Deterministic Policy Gradient and Twin Delayed Deep Deterministic Policy Gradient and evaluate it on various continuous control tasks. KLPER provides promising improvements for deep deterministic continuous control algorithms in terms of sample efficiency, final performance, and stability of the policy during the training.

Improved Deep Deterministic Policy Gradient Algorithm Based on Prioritized Sampling

An Improved Prioritized DDPG Based on Fractional-Order Learning Scheme

Deep Q-Learning with Prioritized Sampling.

Deep Q-learning Sampling Based on Advantages

Off-Policy Correction for Deep Deterministic Policy Gradient Algorithms via Batch Prioritized Experience Replay

Sampling Efficient Deep Reinforcement Learning through Preference-Guided Stochastic Exploration

Deterministic Value-Policy Gradients

Regularly Updated Deterministic Policy Gradient Algorithm

Edge Delayed Deep Deterministic Policy Gradient: efficient continuous control for edge scenarios

Prioritized experience replay based on dynamics priority

Generalizable Policy Improvement Via Reinforcement Sampling (student Abstract)

Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning

Posterior Sampling for Deep Reinforcement Learning

Deep deterministic policy gradient algorithm: A systematic review

Deep Deterministic Policy Gradient Algorithm Based on Convolutional Block Attention for Autonomous Driving.

Network Architecture for Optimizing Deep Deterministic Policy Gradient Algorithms

Prioritized experience replay in path planning via multi-dimensional transition priority fusion

Efficient sample reuse in policy gradients with parameter-based exploration

Sample-Efficient Deep Reinforcement Learning Via Balance Sample

Optimistic Sampling Strategy for Data-Efficient Reinforcement Learning

Asynchronous Episodic Deep Deterministic Policy Gradient: Toward Continuous Control in Computationally Complex Environments