Abstract:In the scenario of class-incremental learning (CIL), deep neural networks have to adapt their model parameters to non-stationary data distributions, e.g., the emergence of new classes over time. However, CIL models are challenged by the well-known catastrophic forgetting phenomenon. Typical methods such as rehearsal-based ones rely on storing exemplars of old classes to mitigate catastrophic forgetting, which limits real-world applications considering memory resources and privacy issues. In this paper, we propose a novel rehearsal-free CIL approach that learns continually via the synergy between two Complementary Learning Subnetworks. Our approach involves jointly optimizing a plastic CNN feature extractor and an analytical feed-forward classifier. The inaccessibility of historical data is tackled by holistically controlling the parameters of a well-trained model, ensuring that the decision boundary learned fits new classes while retaining recognition of previously learned classes. Specifically, the trainable CNN feature extractor provides task-dependent knowledge separately without interference; and the final classifier integrates task-specific knowledge incrementally for decision-making without forgetting. In each CIL session, it accommodates new tasks by attaching a tiny set of declarative parameters to its backbone, in which only one matrix per task or one vector per class is kept for knowledge retention. Extensive experiments on a variety of task sequences show that our method achieves competitive results against state-of-the-art methods, especially in accuracy gain, memory cost, training efficiency, and task-order robustness. Furthermore, to make the non-growing backbone (i.e., a model with limited network capacity) suffice to train on more incoming tasks, a graceful forgetting implementation on previously learned trivial tasks is empirically investigated.

Forget-free Continual Learning with Soft-Winning SubNetworks

Progressive Learning without Forgetting

Continual Learning: Forget-free Winning Subnetworks for Video Representations

Soft-TransFormers for Continual Learning

Learning with Recoverable Forgetting

IF2Net: Innately Forgetting-Free Networks for Continual Learning

Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning

Sub-network Discovery and Soft-masking for Continual Learning of Mixed Tasks

Complementary Learning Subnetworks for Parameter-Efficient Class-Incremental Learning

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

Towards Redundancy-Free Sub-networks in Continual Learning

HyperMask: Adaptive Hypernetwork-based Masks for Continual Learning

Bio-inspired, task-free continual learning through activity regularization

TinySubNets: An efficient and low capacity continual learning strategy

Task-aware Orthogonal Sparse Network for Exploring Shared Knowledge in Continual Learning

Partial Hypernetworks for Continual Learning

Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning

Enhancing Knowledge Transfer for Task Incremental Learning with Data-free Subnetwork.

Continual Learning Via Bit-Level Information Preserving

Incremental Sequential Three-Way Decision Based on Continual Learning Network

Continual Learning with Dependency Preserving Hypernetworks