Abstract:The experience replay mechanism allows agents to use the experiences multiple times. In prior works, the sampling probability of the transitions was adjusted according to their importance. Reassigning sampling probabilities for every transition in the replay buffer after each iteration is highly inefficient. Therefore, experience replay prioritization algorithms recalculate the significance of a transition when the corresponding transition is sampled to gain computational efficiency. However, the importance level of the transitions changes dynamically as the policy and the value function of the agent are updated. In addition, experience replay stores the transitions are generated by the previous policies of the agent that may significantly deviate from the most recent policy of the agent. Higher deviation from the most recent policy of the agent leads to more off-policy updates, which is detrimental for the agent. In this paper, we develop a novel algorithm, Batch Prioritizing Experience Replay via KL Divergence (KLPER), which prioritizes batch of transitions rather than directly prioritizing each transition. Moreover, to reduce the off-policyness of the updates, our algorithm selects one batch among a certain number of batches and forces the agent to learn through the batch that is most likely generated by the most recent policy of the agent. We combine our algorithm with Deep Deterministic Policy Gradient and Twin Delayed Deep Deterministic Policy Gradient and evaluate it on various continuous control tasks. KLPER provides promising improvements for deep deterministic continuous control algorithms in terms of sample efficiency, final performance, and stability of the policy during the training.

Actor Prioritized Experience Replay

High-Value Prioritized Experience Replay For Off-Policy Reinforcement Learning

Prioritized Experience Replay in Multi-Actor-Attention-Critic for Reinforcement Learning

ROER: Regularized Optimal Experience Replay

Attention Loss Adjusted Prioritized Experience Replay

Off-Policy Correction for Deep Deterministic Policy Gradient Algorithms via Batch Prioritized Experience Replay

Ddper - Decentralized Distributed Prioritized Experience Replay.

Prioritized experience replay based on dynamics priority

Directly Attention Loss Adjusted Prioritized Experience Replay.

A priority experience replay actor-critic algorithm using self-attention mechanism for strategy optimization of discrete problems

Investigating the Interplay of Prioritized Replay and Generalization

Fresher Experience Plays a More Important Role in Prioritized Experience Replay

Balanced Prioritized Experience Replay in Off-Policy Reinforcement Learning

TD3 with Composite Forgetting Prioritized Experience Replay

Prioritized Experience Replay

CUER: Corrected Uniform Experience Replay for Off-Policy Continuous Deep Reinforcement Learning Algorithms

Leveraging Efficiency Through Hybrid Prioritized Experience Replay in Door Environment.

Frugal Actor-Critic: Sample Efficient Off-Policy Deep Reinforcement Learning Using Unique Experiences

Multi-Agent Deep Deterministic Policy Gradient Algorithm Based on Classification Experience Replay

Z-Score Experience Replay in Off-Policy Deep Reinforcement Learning

Reverse Experience Replay