Abstract:The experience replay mechanism allows agents to use the experiences multiple times. In prior works, the sampling probability of the transitions was adjusted according to their importance. Reassigning sampling probabilities for every transition in the replay buffer after each iteration is highly inefficient. Therefore, experience replay prioritization algorithms recalculate the significance of a transition when the corresponding transition is sampled to gain computational efficiency. However, the importance level of the transitions changes dynamically as the policy and the value function of the agent are updated. In addition, experience replay stores the transitions are generated by the previous policies of the agent that may significantly deviate from the most recent policy of the agent. Higher deviation from the most recent policy of the agent leads to more off-policy updates, which is detrimental for the agent. In this paper, we develop a novel algorithm, Batch Prioritizing Experience Replay via KL Divergence (KLPER), which prioritizes batch of transitions rather than directly prioritizing each transition. Moreover, to reduce the off-policyness of the updates, our algorithm selects one batch among a certain number of batches and forces the agent to learn through the batch that is most likely generated by the most recent policy of the agent. We combine our algorithm with Deep Deterministic Policy Gradient and Twin Delayed Deep Deterministic Policy Gradient and evaluate it on various continuous control tasks. KLPER provides promising improvements for deep deterministic continuous control algorithms in terms of sample efficiency, final performance, and stability of the policy during the training.

TD3 with Composite Forgetting Prioritized Experience Replay

High-Value Prioritized Experience Replay For Off-Policy Reinforcement Learning

Actor Prioritized Experience Replay

Prioritized Experience Replay in Multi-Actor-Attention-Critic for Reinforcement Learning

Prioritized experience replay based on dynamics priority

Replay-enhanced Continual Reinforcement Learning

Fresher Experience Plays a More Important Role in Prioritized Experience Replay

Leveraging Efficiency Through Hybrid Prioritized Experience Replay in Door Environment.

A priority experience replay actor-critic algorithm using self-attention mechanism for strategy optimization of discrete problems

Prioritized experience replay in path planning via multi-dimensional transition priority fusion

Advances in Experience Replay

Dual Memory Model for Experience-Once Task-Incremental Lifelong Learning.

Multi-Agent Deep Deterministic Policy Gradient Algorithm Based on Classification Experience Replay

Prioritized Experience Replay for Multi-agent Cooperation

Efficient Diversity-based Experience Replay for Deep Reinforcement Learning

A Dynamically Adaptive Approach to Reducing Strategic Interference for Multi-agent Systems

Ddper - Decentralized Distributed Prioritized Experience Replay.

Off-Policy Correction for Deep Deterministic Policy Gradient Algorithms via Batch Prioritized Experience Replay

DDPG with Meta-Learning-Based Experience Replay Separation for Robot Trajectory Planning

ROER: Regularized Optimal Experience Replay

Associative Memory Based Experience Replay for Deep Reinforcement Learning