Abstract:Owing to the superior performances, exemplar-based methods with knowledge distillation (KD) are widely applied in class incremental learning (CIL). However, it suffers from two drawbacks: 1) data imbalance between the old/learned and new classes causes the bias of the new classifier toward the head/new classes and 2) deep neural networks (DNNs) suffer from distribution drift when learning sequence tasks, which results in narrowed feature space and deficient representation of old tasks. For the first problem, we analyze the insufficiency of softmax loss when dealing with the problem of data imbalance in theory and then propose the imbalance softmax (im-softmax) loss to relieve the imbalanced data learning, where we re-scale the output logits to underfit the head/new classes. For another problem, we calibrate the feature space by incremental-adaptive angular margin (IAAM) loss. The new classes form a complete distribution in feature space yet the old are squeezed. To recover the old feature space, we first compute the included angle of normalized features and normalized anchor prototypes, and use the angle distribution to represent the class distribution, then we replenish the old distribution with the deviation from the new. Each anchor prototype is predefined as a learnable vector for a designated class. The proposed im-softmax reduces the bias in the linear classification layer. IAAM rectifies the representation learning, reduces the intra-class distance, and enlarges the inter-class margin. Finally, we seamlessly combine the im-softmax and IAAM in an end-to-end training framework, called the dual balanced class incremental learning (DBL), for further improvements. Experiments demonstrate the proposed method achieves state-of-the-art (SOTA) performances on several benchmarks, such as CIFAR10, CIFAR100, Tiny-ImageNet, and ImageNet-100.

Learning a Unified Classifier Incrementally via Rebalancing

Rethinking Class-Incremental Learning from a Dynamic Imbalanced Learning Perspective

Large Scale Incremental Learning

Discrimination Correction and Balance for Class-Incremental Learning

Maintaining Discrimination and Fairness in Class Incremental Learning

Balanced softmax cross-entropy for incremental learning with and without memory

Multi-Granularity Regularized Re-Balancing for Class Incremental Learning

Dual Balanced Class-Incremental Learning With im-Softmax and Angular Rectification

A self-organizing incremental neural network for imbalance learning

Rebalancing Multi-Label Class-Incremental Learning

Incremental Learning In Online Scenario

Rebalancing network with knowledge stability for class incremental learning

Gradient Reweighting: Towards Imbalanced Class-Incremental Learning

Learning a Dual-Branch Classifier for Class Incremental Learning

Self-Paced Imbalance Rectification for Class Incremental Learning

Overcoming Recency Bias of Normalization Statistics in Continual Learning: Balance and Adaptation

ILCOC: an Incremental Learning Framework Based on Contrastive One-class Classifiers

Boosting Separated Softmax with Discrimination for Class Incremental Learning

Incremental Scene Classification Using Dual Knowledge Distillation and Classifier Discrepancy on Natural and Remote Sensing Images

Class-Incremental Learning: A Survey

Incremental Learning Using Conditional Adversarial Networks