Abstract:Replaying past experiences has proven to be a highly effective approach for averting catastrophic forgetting in supervised continual learning. However, some crucial factors are still largely ignored, making it vulnerable to serious failure, when used as a solution to forgetting in continual reinforcement learning, even in the context of perfect memory where all data of previous tasks are accessible in the current task. On the one hand, since most reinforcement learning algorithms are not invariant to the reward scale, the previously well-learned tasks (with high rewards) may appear to be more salient to the current learning process than the current task (with small initial rewards). This causes the agent to concentrate on those salient tasks at the expense of generality on the current task. On the other hand, offline learning on replayed tasks while learning a new task may induce a distributional shift between the dataset and the learned policy on old tasks, resulting in forgetting. In this paper, we introduce RECALL, a replay-enhanced method that greatly improves the plasticity of existing replay-based methods on new tasks while effectively avoiding the recurrence of catastrophic forgetting in continual reinforcement learning. RECALL leverages adaptive normalization on approximate targets and policy distillation on old tasks to enhance generality and stability, respectively. Extensive experiments on the Continual World benchmark show that RECALL performs significantly better than purely perfect memory replay, and achieves comparable or better overall performance against state-of-the-art continual learning methods.

Distinct replay signatures for prospective decision-making and memory preservation

A unifying account of replay as context-driven memory reactivation

Replay-triggered brain-wide activation in humans

A model of hippocampal replay driven by experience and environmental structure facilitates spatial learning

Prioritizing replay when future goals are unknown

Paradoxical replay can protect contextual task representations from destructive interference when experience is unbalanced

Dual Memory Model for Experience-Once Task-Incremental Lifelong Learning.

Replay-enhanced Continual Reinforcement Learning

Coherent noise enables probabilistic sequence replay in spiking neuronal networks

Reduced Reverse Replay in Anxious Individuals Impairs Reward Prediction

Fast-backward Replay of Sequentially Memorized Items in Humans.

The role of experience in prioritizing hippocampal replay

Neurofeedback training can modulate task-relevant memory replay rate in rats

Replay in human visual cortex is linked to the formation of successor representations and independent of consciousness

Validation of a rodent model of episodic memory replay

Prioritized Sweeping Neural DynaQ with Multiple Predecessors, and Hippocampal Replays

Learning offline: memory replay in biological and artificial reinforcement learning

Exploring the roles of memory replay in targeted memory reactivation and birdsong development: Insights from computational models of complementary learning systems

Inhibitory plasticity supports replay generalization in the hippocampus

Dynamical Modulation of Hippocampal Replay Sequences through Firing Rate Adaptation

Offline orbitofrontal cortex reactivation depends on recency of place-reward changes and coheres with hippocampal replay