Abstract:Deep reinforcement learning has achieved significant success in various domains. However, it still faces a huge challenge when learning multiple tasks in sequence. This is because the interaction in a complex setting involves continual learning that results in the change in data distributions over time. A continual learning system should ensure that the agent acquires new knowledge without forgetting the previous one. However, catastrophic forgetting may occur as the new experience can overwrite previous experience due to limited memory size. The dual experience replay algorithm which retains previous experience is widely applied to reduce forgetting, but it cannot be applied in scalable tasks when the memory size is constrained. To alleviate the constrained by the memory size, we propose a new continual reinforcement learning algorithm called Self-generated Long-term Experience Replay (SLER). Our method is different from the standard dual experience replay algorithm, which uses short-term experience replay to retain current task experience, and the long-term experience replay retains all past tasks’ experience to achieve continual learning. In this paper, we first trained an environment sample model called Experience Replay Mode (ERM) to generate the simulated state sequence of the previous tasks for knowledge retention. Then combined the ERM with the experience of the new task to generate the simulation experience all previous tasks to alleviate forgetting. Our method can effectively decrease the requirement of memory size in multiple tasks, reinforcement learning. We show that our method in StarCraft II and the GridWorld environments performs better than the state-of-the-art deep learning method and achieve a comparable result to the dual experience replay method, which retains the experience of all the tasks.

Parallelized Synchronous Multi-agent Deep Reinforcement Learning with Experience Replay Memory

Synchronous N-Step Method for Independent Q-Learning in Multi-Agent Deep Reinforcement Learning

Lenient Multi-Agent Deep Reinforcement Learning

Dueling Network Architecture for Multi-Agent Deep Deterministic Policy Gradient

Multi-Agent Deep Deterministic Policy Gradient Algorithm Based on Classification Experience Replay

Cooperative multi-agent target searching: a deep reinforcement learning approach based on parallel hindsight experience replay

A Dynamically Adaptive Approach to Reducing Strategic Interference for Multi-agent Systems

AccMER: Accelerating Multi-Agent Experience Replay with Cache Locality-aware Prioritization

Dual Memory Model for Experience-Once Task-Incremental Lifelong Learning.

Leveraging Efficiency Through Hybrid Prioritized Experience Replay in Door Environment.

Experience Selection In Multi-Agent Deep Reinforcement Learning

SLER: Self-generated long-term experience replay for continual reinforcement learning

Deep Reinforcement Learning Approach with Multiple Experience Pools for UAV's Autonomous Motion Planning in Complex Unknown Environments

Efficient Multiagent Policy Optimization Based on Weighted Estimators in Stochastic Cooperative Environments

Weighted Double Deep Multiagent Reinforcement Learning in Stochastic Cooperative Environments

Multi-Input Autonomous Driving Based on Deep Reinforcement Learning with Double Bias Experience Replay

A Novel Multi-Step Q-learning Method to Improve Data Efficiency for Deep Reinforcement Learning.

DDMA: Discrepancy-Driven Multi-agent Reinforcement Learning

Efficient Diversity-based Experience Replay for Deep Reinforcement Learning

Prioritized Experience Replay in Multi-Actor-Attention-Critic for Reinforcement Learning

MARRGM: Learning Framework for Multi-Agent Reinforcement Learning via Reinforcement Recommendation and Group Modification