Abstract:The capability of continuously learning new skills via a sequence of pre-collected offline datasets is desired for an agent. However, consecutively learning a sequence of offline tasks likely leads to the catastrophic forgetting issue under resource-limited scenarios. In this paper, we formulate a new setting, continual offline reinforcement learning (CORL), where an agent learns a sequence of offline reinforcement learning tasks and pursues good performance on all learned tasks with a small replay buffer without exploring any of the environments of all the sequential tasks. For consistently learning on all sequential tasks, an agent requires acquiring new knowledge and meanwhile preserving old knowledge in an offline manner. To this end, we introduced continual learning algorithms and experimentally found experience replay (ER) to be the most suitable algorithm for the CORL problem. However, we observe that introducing ER into CORL encounters a new distribution shift problem: the mismatch between the experiences in the replay buffer and trajectories from the learned policy. To address such an issue, we propose a new model-based experience selection (MBES) scheme to build the replay buffer, where a transition model is learned to approximate the state distribution. This model is used to bridge the distribution bias between the replay buffer and the learned model by filtering the data from offline data that most closely resembles the learned model for storage. Moreover, in order to enhance the ability on learning new tasks, we retrofit the experience replay method with a new dual behavior cloning (DBC) architecture to avoid the disturbance of behavior-cloning loss on the Q-learning process. In general, we call our algorithm offline experience replay (OER). Extensive experiments demonstrate that our OER method outperforms SOTA baselines in widely-used Mujoco environments.

Parallel Curriculum Experience Replay in Distributed Reinforcement Learning.

Robot Simulation and Reinforcement Learning Training Platform Based on Distributed Architecture.

Efficient Diversity-based Experience Replay for Deep Reinforcement Learning

Ddper - Decentralized Distributed Prioritized Experience Replay.

Parallel learner: A practical deep reinforcement learning framework for multi-scenario games

SLER: Self-generated long-term experience replay for continual reinforcement learning

Advances in Experience Replay

Cooperative multi-agent target searching: a deep reinforcement learning approach based on parallel hindsight experience replay

Distributed Deep Reinforcement Learning: A Survey and a Multi-player Multi-agent Learning Toolbox

Joint Action Representation and Prioritized Experience Replay for Reinforcement Learning in Large Discrete Action Spaces.

Experience Selection In Multi-Agent Deep Reinforcement Learning

ACDER: Augmented Curiosity-Driven Experience Replay

Self-Paced Prioritized Curriculum Learning with Coverage Penalty in Deep Reinforcement Learning.

rSoccer: A Framework for Studying Reinforcement Learning in Small and Very Small Size Robot Soccer

Deep reinforcement learning algorithm based on multi-agent parallelism and its application in game environment

Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replay

Asynchronous Curriculum Experience Replay: A Deep Reinforcement Learning Approach for UAV Autonomous Motion Control in Unknown Dynamic Environments

Replay-enhanced Continual Reinforcement Learning

A Dual Memory Structure for Efficient Use of Replay Memory in Deep Reinforcement Learning

Soft Hindsight Experience Replay

OER: Offline Experience Replay for Continual Offline Reinforcement Learning