Abstract:Although the concept of Catastrophic Forgetting (CF) is straightforward in Class Incremental Learning (CIL), the causes of CF in models are still vague. In this paper, by introducing the metrics of representation quality, we investigate and analyze systematically, yet quantitatively, the three primary causes for the catastrophic forgetting of model in CIL, namely Intra-phase Forgetting (IpF), Inter-phase Confusion (IpC) and Classifier Deviation (CD). The (i) IpF happens when the learner fails to correctly align the same-phase data as training proceeds, (ii) IpC happens when the learner confuses the current-phase data with the previous, and (iii) CD happens when the old classifier deviates from the current representation space.Â By studying extensively, we have discovered that the current distillation process can effectively address the IpF issue, and NN-based techniques have brought some alleviation to CD. Nevertheless, the IpC dilemma still awaits further exploration. To spark a more in-depth inquiry into this highly relevant question, we propose a simple yet effective framework, \textbf{C}ontrastive \textbf{C}lass \textbf{C}oncentration for \textbf{CIL}~(C4IL). It utilizes the benefit of distillation and contrastive learning, producing a representation distribution that is more cohesive within the same class and clearly different across different classes, in order to reduce representation overlapping throughout the various training phases and address the issue of IpC. Quantitative experiments showcase the effectiveness of our framework: it outperforms the baseline methods iCaRL by 10\% in top-1 accuracy at the final training phase, and the average accuracy is competitive with the up-to-date method RMM. Qualitative results also demonstrate that C4IL significantly lowers the probability of inter-phase confusion and generates a more compact representation distribution which alleviates the IpC problem.

TCP: Triplet Contrastive-relationship Preserving for Class-Incremental Learning

Connection-Based Knowledge Transfer for Class Incremental Learning

Class Incremental Learning with Deep Contrastive Learning and Attention Distillation

Rethinking Class-Incremental Learning from a Dynamic Imbalanced Learning Perspective

Memory Efficient Class-Incremental Learning for Image Classification

Maintaining Discrimination and Fairness in Class Incremental Learning

Class-Incremental Learning: A Survey

On Representation-Level Forgetting in Class Incremental Learning: What's the Bottleneck?

Defying Imbalanced Forgetting in Class Incremental Learning

Inherit With Distillation and Evolve With Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory

Complementary Learning Subnetworks for Parameter-Efficient Class-Incremental Learning

Alleviate Representation Overlapping in Class Incremental Learning by Contrastive Class Concentration

Mitigating Catastrophic Forgetting in Task-Incremental Continual Learning with Adaptive Classification Criterion

Federated Class-Incremental Learning with Prompting

Resolving Task Confusion in Dynamic Expansion Architectures for Class Incremental Learning

Class incremental learning of remote sensing images based on class similarity distillation

Contrastive-based Bi-update Class Incremental Model and Its Application in Fault Diagnosis

Rebalancing network with knowledge stability for class incremental learning

Class-Incremental Learning Based on Big Dataset Pre-Trained Models

CLUE: Consolidating Learned and Undergoing Experience in Domain-Incremental Classification

PASS++: A Dual Bias Reduction Framework for Non-Exemplar Class-Incremental Learning