Abstract:The capability of continuously learning new skills via a sequence of pre-collected offline datasets is desired for an agent. However, consecutively learning a sequence of offline tasks likely leads to the catastrophic forgetting issue under resource-limited scenarios. In this paper, we formulate a new setting, continual offline reinforcement learning (CORL), where an agent learns a sequence of offline reinforcement learning tasks and pursues good performance on all learned tasks with a small replay buffer without exploring any of the environments of all the sequential tasks. For consistently learning on all sequential tasks, an agent requires acquiring new knowledge and meanwhile preserving old knowledge in an offline manner. To this end, we introduced continual learning algorithms and experimentally found experience replay (ER) to be the most suitable algorithm for the CORL problem. However, we observe that introducing ER into CORL encounters a new distribution shift problem: the mismatch between the experiences in the replay buffer and trajectories from the learned policy. To address such an issue, we propose a new model-based experience selection (MBES) scheme to build the replay buffer, where a transition model is learned to approximate the state distribution. This model is used to bridge the distribution bias between the replay buffer and the learned model by filtering the data from offline data that most closely resembles the learned model for storage. Moreover, in order to enhance the ability on learning new tasks, we retrofit the experience replay method with a new dual behavior cloning (DBC) architecture to avoid the disturbance of behavior-cloning loss on the Q-learning process. In general, we call our algorithm offline experience replay (OER). Extensive experiments demonstrate that our OER method outperforms SOTA baselines in widely-used Mujoco environments.

ER-FSL: Experience Replay with Feature Subspace Learning for Online Continual Learning

Progressive Learning without Forgetting

Continual Learning with Strong Experience Replay

Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning

Contrastive Correlation Preserving Replay for Online Continual Learning

Squeezing More Past Knowledge for Online Class-Incremental Continual Learning

Adaptive Memory Replay for Continual Learning

AdaER: An Adaptive Experience Replay Approach for Continual Lifelong Learning

OER: Offline Experience Replay for Continual Offline Reinforcement Learning

Principal Gradient Direction and Confidence Reservoir Sampling for Continual Learning

Adaptive online continual multi-view learning

SLER: Self-generated long-term experience replay for continual reinforcement learning

UER: A Heuristic Bias Addressing Approach for Online Continual Learning

Saliency-Guided Hidden Associative Replay for Continual Learning

Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System

Dual Memory Model for Experience-Once Task-Incremental Lifelong Learning.

Effective Data Selection and Replay for Unsupervised Continual Learning

Replay-enhanced Continual Reinforcement Learning

Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learning

May the Forgetting Be with You: Alternate Replay for Learning with Noisy Labels

Representation Space Maintenance: Against Forgetting in Continual Learning