Abstract:Deep neural networks (DNNs) are prone to the notorious catastrophic forgetting problem when learning new tasks incrementally. Class-incremental learning (CIL) is a promising solution to tackle the challenge and learn new classes while not forgetting old ones. Existing CIL approaches adopted stored representative exemplars or complex generative models to achieve good performance. However, storing data from previous tasks causes memory or privacy issues, and the training of generative models is unstable and inefficient. This paper proposes a method based on multi-granularity knowledge distillation and prototype consistency regularization (MDPCR) that performs well even when the previous training data is unavailable. First, we propose to design knowledge distillation losses in the deep feature space to constrain the incremental model trained on the new data. Thereby, multi-granularity is captured from three aspects: by distilling multi-scale self-attentive features, the feature similarity probability, and global features to maximize the retention of previous knowledge, effectively alleviating catastrophic forgetting. Conversely, we preserve the prototype of each old class and employ prototype consistency regularization (PCR) to ensure that the old prototypes and semantically enhanced prototypes produce consistent prediction, which excels in enhancing the robustness of old prototypes and reduces the classification bias. Extensive experiments on three CIL benchmark datasets confirm that MDPCR performs significantly better over exemplar-free methods and outperforms typical exemplar-based approaches.

Class Incremental Learning with Deep Contrastive Learning and Attention Distillation

A Class Incremental Learning Method with Weight Re-Initialization and Multiple Distillation for Image Classification

Maintaining Discrimination and Fairness in Class Incremental Learning

Inherit With Distillation and Evolve With Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory

Contrastive-based Bi-update Class Incremental Model and Its Application in Fault Diagnosis

Class incremental learning of remote sensing images based on class similarity distillation

Class-Incremental Learning: A Survey

Mitigating Catastrophic Forgetting in Task-Incremental Continual Learning with Adaptive Classification Criterion

Model Behavior Preserving for Class-Incremental Learning

Distilling Causal Effect of Data in Class-Incremental Learning

TCP: Triplet Contrastive-relationship Preserving for Class-Incremental Learning

Class-Incremental Learning Based on Big Dataset Pre-Trained Models

Fine-Grained Knowledge Selection and Restoration for Non-Exemplar Class Incremental Learning

Discriminative Distillation to Reduce Class Confusion in Continual Learning

Incremental Scene Classification Using Dual Knowledge Distillation and Classifier Discrepancy on Natural and Remote Sensing Images

Reducing Catastrophic Forgetting in Online Class Incremental Learning Using Self-Distillation

Incremental Learning Via Rate Reduction

Rethinking Class-Incremental Learning from a Dynamic Imbalanced Learning Perspective

Multi-granularity knowledge distillation and prototype consistency regularization for class-incremental learning

Bayesian Learning-driven Prototypical Contrastive Loss for Class-Incremental Learning