Abstract:Artificial neural networks, especially recent diffusion-based models, have shown remarkable superiority in gaming, control, and QA systems, where the training tasks' datasets are usually static. However, in real-world applications, such as robotic control of reinforcement learning (RL), the tasks are changing, and new tasks arise in a sequential order. This situation poses the new challenge of plasticity-stability trade-off for training an agent who can adapt to task changes and retain acquired knowledge. In view of this, we propose a rehearsal-based continual diffusion model, called Continual Diffuser (CoD), to endow the diffuser with the capabilities of quick adaptation (plasticity) and lasting retention (stability). Specifically, we first construct an offline benchmark that contains 90 tasks from multiple domains. Then, we train the CoD on each task with sequential modeling and conditional generation for making decisions. Next, we preserve a small portion of previous datasets as the rehearsal buffer and replay it to retain the acquired knowledge. Extensive experiments on a series of tasks show CoD can achieve a promising plasticity-stability trade-off and outperform existing diffusion-based methods and other representative baselines on most tasks.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to address the plasticity-stability trade-off encountered in Continual Offline Reinforcement Learning (CORL). Specifically: 1. **Research Background**: - Current artificial neural networks, especially diffusion models, perform well in decision-making scenarios (such as games, robotic control, etc.), but these tasks' datasets are usually static. - In real-world applications, such as reinforcement learning in robotic control, tasks continuously change, and new tasks appear sequentially. This poses a challenge for training an agent that can both adapt to task changes and retain learned knowledge. 2. **Main Problem**: - Current methods often fail to continue learning effectively when faced with new tasks because they struggle with the conflict between storing historical knowledge (stability) and adapting to environmental changes (plasticity). - Existing diffusion models, although performing well on specific tasks, are prone to catastrophic forgetting in continuous task environments. 3. **Proposed Method**: - The paper proposes a Continual Diffuser (CoD) based on experience rehearsal, aiming to enable diffusion models to quickly adapt to new tasks (plasticity) while retaining existing knowledge (stability) when handling continuous tasks. - CoD makes decisions by performing sequence modeling and conditional generation on each task and retains a small portion of previous tasks' data as a rehearsal buffer, periodically rehearsing to maintain existing knowledge. 4. **Experimental Validation**: - The paper constructs an offline benchmark set containing 90 tasks and conducts extensive experiments across a series of tasks, demonstrating that CoD excels in balancing plasticity and stability and outperforms existing diffusion models and other representative baseline methods in most tasks.

Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsal

Instructed Diffuser with Temporal Condition Guidance for Offline Reinforcement Learning

Stable Continual Reinforcement Learning via Diffusion-based Trajectory Replay

Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replay

Solving Continual Offline RL through Selective Weights Activation on Aligned Spaces

Dynamics-Adaptive Continual Reinforcement Learning Via Progressive Contextualization.

Contrastive Diffuser: Planning Towards High Return States via Contrastive Learning

Experience Consistency Distillation Continual Reinforcement Learning for Robotic Manipulation Tasks

MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL

Diffusion Reward: Learning Rewards via Conditional Video Diffusion

Diffskill: Improving Reinforcement Learning Through Diffusion-Based Skill Denoiser for Robotic Manipulation

Replay-enhanced Continual Reinforcement Learning

Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning

Long-Horizon Rollout via Dynamics Diffusion for Offline Reinforcement Learning

Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control

Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning

Diffusion Actor-Critic with Entropy Regulator

Off-dynamics Conditional Diffusion Planners

Training Diffusion Models with Reinforcement Learning

Boosting Continuous Control with Consistency Policy

Continual Learning Using World Models for Pseudo-Rehearsal