Abstract:In continual learning, a model learns incrementally over time while minimizing interference between old and new tasks. One of the most widely used approaches in continual learning is referred to as replay. Replay methods support interleaved learning by storing past experiences in a replay buffer. Although there are methods for selectively constructing the buffer and reprocessing its contents, there is limited exploration of the problem of selectively retrieving samples from the buffer. Current solutions have been tested in limited settings and, more importantly, in isolation. Existing work has also not explored the impact of duplicate replays on performance. In this work, we propose a framework for evaluating selective retrieval strategies, categorized by simple, independent class- and sample-selective primitives. We evaluated several combinations of existing strategies for selective retrieval and present their performances. Furthermore, we propose a set of strategies to prevent duplicate replays and explore whether new samples with low loss values can be learned without replay. In an effort to match our problem setting to a realistic continual learning pipeline, we restrict our experiments to a setting involving a large, pre-trained, open vocabulary object detection model, which is fully fine-tuned on a sequence of 15 datasets.

What problem does this paper attempt to address?

This paper discusses how to optimize the replay strategy in continual learning to effectively prevent interference between old and new tasks. Continual learning is a method that allows models to gradually learn new knowledge over time while minimizing the impact on previous tasks. The replay strategy supports interleaved learning by storing past experience (data) and mixing old task data into new task data. Current replay methods mainly focus on how to selectively construct and process the content of the replay buffer. However, research on selectively retrieving samples from the buffer is limited, and these methods are evaluated in limited settings without directly comparing the performance of different algorithms. In addition, there is still insufficient research on how to avoid repeated replay and whether all new samples need to be replayed. The paper proposes a framework to evaluate selective retrieval strategies and categorizes these strategies into simple and independent primitives based on category or sample. The paper also proposes strategies to prevent repeated replay and explores whether new samples can be learned without replay when the loss value is low. The experiments are conducted on a large pre-trained open vocabulary object detection model, which undergoes full fine-tuning on 15 dataset sequences to match real-world continual learning environments. The study found that the performance of the two simplest primitives as independent solutions is superior to other algorithms, including combinations of these primitives. Avoiding repeated replay is necessary for almost all algorithms, but to some extent, it is shared. Furthermore, reducing the impact of replay on previous tasks when the loss of new samples is below a threshold leads to forgetting of the previous tasks. In summary, the main contributions of the paper include: 1. Comparison of selective retrieval algorithms, demonstrating that certain algorithms are better than others, including combinations of themselves. 2. Comparison of different non-repetitive replay intervals, indicating that repeated replay should be prevented for each downstream dataset. 3. Showing that replaying only high-loss new samples leads to a rapid decline in overall performance. 4. Analyzing the sample and category distributions generated by the best algorithm and the forgetting dynamics based on dataset characteristics. The paper also discusses related work, including the classification of continual learning, model expansion and replay methods, as well as specific algorithms for selective replay. Finally, the paper provides experimental settings, methods, results, and analysis, emphasizing the complexity of selective retrieval strategies and the importance of balancing selectivity and diversity.

Watch Your Step: Optimal Retrieval for Continual Learning at Scale

RECL: Responsive Resource-Efficient Continuous Learning for Video Analytics

Improving Replay Sample Selection and Storage for Less Forgetting in Continual Learning

Adaptive Memory Replay for Continual Learning

Replay-enhanced Continual Reinforcement Learning

A Benchmark and Empirical Analysis for Replay Strategies in Continual Learning

Continual Learning: Less Forgetting, More OOD Generalization via Adaptive Contrastive Replay

Continual Recognition with Adaptive Memory Update.

TEAL: New Selection Strategy for Small Buffers in Experience Replay Class Incremental Learning

SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language Models

The Effectiveness of Memory Replay in Large Scale Continual Learning

Saliency-Guided Hidden Associative Replay for Continual Learning

A rehearsal framework for computational efficiency in online continual learning

Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes

Continual Learning with Strong Experience Replay

CORE: Mitigating Catastrophic Forgetting in Continual Learning through Cognitive Replay

Principal Gradient Direction and Confidence Reservoir Sampling for Continual Learning

Learn the Time to Learn: Replay Scheduling in Continual Learning

Relational Experience Replay: Continual Learning by Adaptively Tuning Task-wise Relationship

Looking Through the Past: Better Knowledge Retention for Generative Replay in Continual Learning

Saliency Guided Experience Packing for Replay in Continual Learning