Abstract:Although neural networks have been used extensively in pattern recognition scenarios, the pre-acquisition of datasets is still challenging. In most pattern recognition areas, preparing a training dataset that covers all data domains is difficult. Incremental learning was proposed to update neural networks in an online manner, but the catastrophic forgetting issue still needs to be studied. Class-incremental learning is one of the most challenging incremental learning contexts; it trains a unified model to classify all incrementally arriving classes learned thus far equally. Prior studies on class-incremental learning favor model stability over plasticity to realize old knowledge reservation and prevent catastrophic forgetting. Consequently, the model's plasticity is omitted, leading to difficult generalization on new data. We propose a novel distillation-based method named Hyper-feature Aggregation and Relaxed Distillation (HARD) to realize balanced optimization of old and new knowledge. The aggregation of features is proposed to capture the global semantics while maintaining the diversity of the feature distribution after promoting representations of exemplars to higher dimensions. The proposed algorithm also introduces a relaxed restriction in the hyper-feature space to conditions the hyper-feature space through a normalized comparison of the relation matrices. Following generalization on more classes, the model is encouraged to rebuild the feature distribution when meeting new classes and to fine-tune the feature space to realize more distinct interclass boundaries. Extensive experiments were conducted on two benchmark datasets, and consistent improvements under diverse experimental settings demonstrated the effectiveness of the proposed approach.

DeepCollaboration: Collaborative Generative and Discriminative Models for Class Incremental Learning

Generative Feature Replay For Class-Incremental Learning

DDGR: Continual Learning with Deep Diffusion-based Generative Replay.

Intra-Model Collaborative Learning of Neural Networks.

Incremental Learning Using Conditional Adversarial Networks

Autonomous Generative Feature Replay for Non-Exemplar Class-Incremental Learning

General Incremental Learning with Domain-aware Categorical Representations

Incremental Classifier Learning with Generative Adversarial Networks

Deep Collaborative Learning for Visual Recognition.

Large Scale Incremental Learning

Class-Incremental Mixture of Gaussians for Deep Continual Learning

Efficient Non-Exemplar Class-Incremental Learning with Retrospective Feature Synthesis

Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning

Memory-Free Generative Replay For Class-Incremental Learning

Maintaining Discrimination and Fairness in Class Incremental Learning

Class-Incremental Learning using Diffusion Model for Distillation and Replay

Bypassing Logits Bias in Online Class-Incremental Learning with a Generative Framework

Deep Class Incremental Learning from Decentralized Data

Forget but Recall: Incremental Latent Rectification in Continual Learning

Hyper-feature aggregation and relaxed distillation for class incremental learning

Harnessing Neural Unit Dynamics for Effective and Scalable Class-Incremental Learning