Abstract:Continual learning (CL) provides a framework for training models in ever-evolving environments. Although re-occurrence of previously seen objects or tasks is common in real-world problems, the concept of repetition in the data stream is not often considered in standard benchmarks for CL. Unlike with the rehearsal mechanism in buffer-based strategies, where sample repetition is controlled by the strategy, repetition in the data stream naturally stems from the environment. This report provides a summary of the CLVision challenge at CVPR 2023, which focused on the topic of repetition in class-incremental learning. The report initially outlines the challenge objective and then describes three solutions proposed by finalist teams that aim to effectively exploit the repetition in the stream to learn continually. The experimental results from the challenge highlight the effectiveness of ensemble-based solutions that employ multiple versions of similar modules, each trained on different but overlapping subsets of classes. This report underscores the transformative potential of taking a different perspective in CL by employing repetition in the data stream to foster innovative strategy design.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper attempts to address the issue of how to leverage the phenomenon of repetition in data streams to improve model performance in Continual Learning (CL). Specifically, the paper focuses on how to effectively utilize repeated samples in class-incremental learning to enhance knowledge transfer and reduce forgetting. ### Background and Motivation In traditional machine learning, models are typically trained on static datasets, which assume that samples are independently and identically distributed (IID). However, in the real world, this assumption is often violated, and various forms of data shifts occur in data streams. To cope with these shifts, continual learning provides a framework that simulates an "endless" learning environment. In continual learning, the model encounters a potentially infinite data stream, which is presented to the model in a non-IID manner, partially and gradually. ### Importance of the Repetition Phenomenon Although the phenomenon of repetition is very common in the real world, it is often not adequately considered in standard continual learning benchmarks. Traditional continual learning research typically uses "academic scenarios" where tasks, domains, or classes appear in a strict sequence without the repetition of previously seen concepts. However, in real-world problems, previously encountered concepts reappear in different ways, which can lead to improved understanding of the concepts. ### Research Objectives The main objective of the paper is to explore the role of the repetition phenomenon in continual learning and to design strategies that can effectively utilize the repetition phenomenon in data streams. Specifically, participants need to design strategies that can leverage the repetition phenomenon in data streams to enhance knowledge transfer and reduce forgetting without storing the original samples. ### Main Contributions 1. **Proposed New Challenges**: By generating data streams that include the repetition phenomenon, a series of benchmarks were designed to evaluate the performance of different strategies in handling the repetition phenomenon. 2. **Provided Solutions**: Introduced the solutions proposed by three finalist teams, which effectively utilized the repetition phenomenon in data streams through various methods such as ensemble learning and supervised contrastive learning. 3. **Experimental Results**: Demonstrated experimental results that proved the effectiveness of ensemble-based methods in handling data streams that include the repetition phenomenon. ### Conclusion The paper emphasizes the importance of considering the repetition phenomenon in continual learning and points out that by leveraging the repetition phenomenon in data streams, more efficient strategies can be developed, especially in environments where pattern repetition follows certain rules. This research provides new perspectives and directions for the design of future continual learning strategies.

Continual Learning in the Presence of Repetition

Batch-level Experience Replay with Review for Continual Learning

RepCL: Exploring Effective Representation for Continual Text Classification

CVPR 2020 Continual Learning in Computer Vision Competition: Approaches, Results, Current Challenges and Future Directions

From MNIST to ImageNet and back: benchmarking continual curriculum learning

Adaptive Memory Replay for Continual Learning

Continual Learning with Strong Experience Replay

A Closer Look at Rehearsal-Free Continual Learning

Don't Stop Learning: Towards Continual Learning for the CLIP Model

Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes

Brain-inspired Continuous Learning: Technology, Application and Future

Reinforced Continual Learning

Dissecting Continual Learning a Structural and Data Analysis

Watch Your Step: Optimal Retrieval for Continual Learning at Scale

Expanding continual few-shot learning benchmarks to include recognition of specific instances

Continual Learning: Applications and the Road Forward

The CLEAR Benchmark: Continual LEArning on Real-World Imagery

CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks

Continual Recognition with Adaptive Memory Update.

A baseline on continual learning methods for video action recognition