Abstract:Deep models, e.g., CNNs and Vision Transformers, have achieved impressive achievements in many vision tasks in the closed world. However, novel classes emerge from time to time in our ever-changing world, requiring a learning system to acquire new knowledge continually. Class-Incremental Learning (CIL) enables the learner to incorporate the knowledge of new classes incrementally and build a universal classifier among all seen classes. Correspondingly, when directly training the model with new class instances, a fatal problem occurs -- the model tends to catastrophically forget the characteristics of former ones, and its performance drastically degrades. There have been numerous efforts to tackle catastrophic forgetting in the machine learning community. In this paper, we survey comprehensively recent advances in class-incremental learning and summarize these methods from several aspects. We also provide a rigorous and unified evaluation of 17 methods in benchmark image classification tasks to find out the characteristics of different algorithms empirically. Furthermore, we notice that the current comparison protocol ignores the influence of memory budget in model storage, which may result in unfair comparison and biased results. Hence, we advocate fair comparison by aligning the memory budget in evaluation, as well as several memory-agnostic performance measures. The source code is available at <a class="link-external link-https" href="https://github.com/zhoudw-zdw/CIL_Survey/" rel="external noopener nofollow">this https URL</a>

What problem does this paper attempt to address?

### Problems Addressed by the Paper This paper primarily explores the key issues in Class-Incremental Learning (CIL) and presents the latest advancements and solutions. #### Core Issues 1. **Catastrophic Forgetting**: In a constantly changing world, new categories continuously emerge, requiring learning systems to continuously acquire new knowledge. However, when training a model directly with instances of new categories, the model often forgets previously learned knowledge, leading to a sharp decline in performance. Effectively resisting this catastrophic forgetting has become a core issue in building CIL models. #### Research Background - Deep learning (such as CNN and Vision Transformers) has achieved significant results in many visual tasks in a closed world. - In an open world, data usually appears in a stream, and storage is limited or there are privacy issues, thus requiring models to be incrementally updated. - Class-Incremental Learning aims to enable models to gradually build a universal classifier that covers all seen categories. #### Summary of Contributions - Provides a comprehensive review of CIL, including problem definitions, benchmark datasets, and different types of CIL methods. - Offers a rigorous unified comparison on multiple benchmark datasets, covering traditional CNN methods and modern ViT methods. - Emphasizes an important factor in evaluating CIL models—memory budget, and advocates for fair comparisons under the same budget. - Provides comprehensive evaluations under specific budgets and several new performance metrics. #### Method Classifications - Data Replay: Prevents forgetting by saving and revisiting previous instances. - Data Regularization: Uses previous instances as indicators to regularize model updates. - Dynamic Networks: Expands network structures to enhance representation capabilities. - Parameter Regularization: Regularizes model parameters to prevent them from deviating. - Knowledge Distillation: Constructs mappings between incremental models to prevent forgetting. - Model Rectify: Reduces biased predictions of incremental learners. - Template-Based Classification: Transforms inference into query-template matching. #### Development Trends - Before 2021, knowledge distillation and data replay dominated research directions. - After 2021, model rectification and dynamic networks became research hotspots. Through these reviews and classifications, the paper aims to further advance the field of Class-Incremental Learning and provide practical guidance and benchmark results.

Class-Incremental Learning: A Survey

Class-Incremental Learning: Survey and Performance Evaluation on Image Classification

Memory Efficient Class-Incremental Learning for Image Classification

Semantic Knowledge Guided Class-Incremental Learning

Class-Incremental Learning with Topological Schemas of Memory Spaces

Class Incremental Learning with Important and Diverse Memory.

Large Scale Incremental Learning

Self-Supervised Class Incremental Learning

Class Incremental Learning with Self-Supervised Pre-Training and Prototype Learning

DiffClass: Diffusion-Based Class Incremental Learning

Class-incremental Learning for Time Series: Benchmark and Evaluation

Complementary Learning Subnetworks for Parameter-Efficient Class-Incremental Learning

Curiosity-Driven Class-Incremental Learning Via Adaptive Sample Selection

A Robust and Anti-Forgettiable Model for Class-Incremental Learning

Class-Incremental Learning with Strong Pre-trained Models

Online continual learning in image classification: An empirical survey

Class Incremental Learning with Deep Contrastive Learning and Attention Distillation

Reformulating Classification As Image-Class Matching for Class Incremental Learning

Class-Incremental Learning Based on Anomaly Detection

Class-Incremental Exemplar Compression for Class-Incremental Learning

Class Incremental Learning with Task-Specific Batch Normalization and Out-of-Distribution Detection