Abstract:Owing to the superior performances, exemplar-based methods with knowledge distillation (KD) are widely applied in class incremental learning (CIL). However, it suffers from two drawbacks: 1) data imbalance between the old/learned and new classes causes the bias of the new classifier toward the head/new classes and 2) deep neural networks (DNNs) suffer from distribution drift when learning sequence tasks, which results in narrowed feature space and deficient representation of old tasks. For the first problem, we analyze the insufficiency of softmax loss when dealing with the problem of data imbalance in theory and then propose the imbalance softmax (im-softmax) loss to relieve the imbalanced data learning, where we re-scale the output logits to underfit the head/new classes. For another problem, we calibrate the feature space by incremental-adaptive angular margin (IAAM) loss. The new classes form a complete distribution in feature space yet the old are squeezed. To recover the old feature space, we first compute the included angle of normalized features and normalized anchor prototypes, and use the angle distribution to represent the class distribution, then we replenish the old distribution with the deviation from the new. Each anchor prototype is predefined as a learnable vector for a designated class. The proposed im-softmax reduces the bias in the linear classification layer. IAAM rectifies the representation learning, reduces the intra-class distance, and enlarges the inter-class margin. Finally, we seamlessly combine the im-softmax and IAAM in an end-to-end training framework, called the dual balanced class incremental learning (DBL), for further improvements. Experiments demonstrate the proposed method achieves state-of-the-art (SOTA) performances on several benchmarks, such as CIFAR10, CIFAR100, Tiny-ImageNet, and ImageNet-100.

Supplementary Material forCo-Transport for Class-Incremental Learning

Co-Transport for Class-Incremental Learning

Continual Learning with Optimal Transport based Mixture Model

A Cognition-Driven Framework for Few-Shot Class-Incremental Learning

DS-AL: A Dual-Stream Analytic Learning for Exemplar-Free Class-Incremental Learning

CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One

Preserve Knowledge with Auxiliary Feature Extractor for Class Incremental Learning

OrCo: Towards Better Generalization via Orthogonality and Contrast for Few-Shot Class-Incremental Learning

TCP: Triplet Contrastive-relationship Preserving for Class-Incremental Learning

I2CANSAY:Inter-Class Analogical Augmentation and Intra-Class Significance Analysis for Non-Exemplar Online Task-Free Continual Learning

Class-Incremental Learning: A Survey

Harnessing Neural Unit Dynamics for Effective and Scalable Class-Incremental Learning

Class-Incremental Learning with Strong Pre-trained Models

Class-Incremental Exemplar Compression for Class-Incremental Learning

A class-incremental learning approach for learning feature-compatible embeddings

Towards Non-Exemplar Semi-Supervised Class-Incremental Learning

Heterogeneous Forgetting Compensation for Class-Incremental Learning

Memory Efficient Class-Incremental Learning for Image Classification

Alleviate Representation Overlapping in Class Incremental Learning by Contrastive Class Concentration

Dual Balanced Class-Incremental Learning With im-Softmax and Angular Rectification

Connection-Based Knowledge Transfer for Class Incremental Learning