Abstract:In the scenario of class-incremental learning (CIL), deep neural networks have to adapt their model parameters to non-stationary data distributions, e.g., the emergence of new classes over time. However, CIL models are challenged by the well-known catastrophic forgetting phenomenon. Typical methods such as rehearsal-based ones rely on storing exemplars of old classes to mitigate catastrophic forgetting, which limits real-world applications considering memory resources and privacy issues. In this paper, we propose a novel rehearsal-free CIL approach that learns continually via the synergy between two Complementary Learning Subnetworks. Our approach involves jointly optimizing a plastic CNN feature extractor and an analytical feed-forward classifier. The inaccessibility of historical data is tackled by holistically controlling the parameters of a well-trained model, ensuring that the decision boundary learned fits new classes while retaining recognition of previously learned classes. Specifically, the trainable CNN feature extractor provides task-dependent knowledge separately without interference; and the final classifier integrates task-specific knowledge incrementally for decision-making without forgetting. In each CIL session, it accommodates new tasks by attaching a tiny set of declarative parameters to its backbone, in which only one matrix per task or one vector per class is kept for knowledge retention. Extensive experiments on a variety of task sequences show that our method achieves competitive results against state-of-the-art methods, especially in accuracy gain, memory cost, training efficiency, and task-order robustness. Furthermore, to make the non-growing backbone (i.e., a model with limited network capacity) suffice to train on more incoming tasks, a graceful forgetting implementation on previously learned trivial tasks is empirically investigated.

Training Networks in Null Space of Feature Covariance for Continual Learning

Progressive Learning without Forgetting

Balancing Stability and Plasticity Through Advanced Null Space in Continual Learning

PNSP: Overcoming Catastrophic Forgetting Using Primary Null Space Projection in Continual Learning

Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning

Uncertainty Estimation With Neural Processes for Meta-Continual Learning

Task Agnostic Continual Learning via Meta Learning

Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting

Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding

Create and Find Flatness: Building Flat Training Spaces in Advance for Continual Learning

Sparsity and Heterogeneous Dropout for Continual Learning in the Null Space of Neural Activations

Reinforced Continual Learning

Gradient Correlation Subspace Learning against Catastrophic Forgetting

Wide Neural Networks Forget Less Catastrophically

CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One

Complementary Learning Subnetworks for Parameter-Efficient Class-Incremental Learning

Continual Learning by Modeling Intra-Class Variation

Lifelong Learning Process: Self-Memory Supervising and Dynamically Growing Networks

Introducing Common Null Space of Gradients for Gradient Projection Methods in Continual Learning

On the Convergence of Continual Learning with Adaptive Methods

Orthogonal Gradient Descent for Continual Learning