Abstract:In the scenario of class-incremental learning (CIL), deep neural networks have to adapt their model parameters to non-stationary data distributions, e.g., the emergence of new classes over time. However, CIL models are challenged by the well-known catastrophic forgetting phenomenon. Typical methods such as rehearsal-based ones rely on storing exemplars of old classes to mitigate catastrophic forgetting, which limits real-world applications considering memory resources and privacy issues. In this paper, we propose a novel rehearsal-free CIL approach that learns continually via the synergy between two Complementary Learning Subnetworks. Our approach involves jointly optimizing a plastic CNN feature extractor and an analytical feed-forward classifier. The inaccessibility of historical data is tackled by holistically controlling the parameters of a well-trained model, ensuring that the decision boundary learned fits new classes while retaining recognition of previously learned classes. Specifically, the trainable CNN feature extractor provides task-dependent knowledge separately without interference; and the final classifier integrates task-specific knowledge incrementally for decision-making without forgetting. In each CIL session, it accommodates new tasks by attaching a tiny set of declarative parameters to its backbone, in which only one matrix per task or one vector per class is kept for knowledge retention. Extensive experiments on a variety of task sequences show that our method achieves competitive results against state-of-the-art methods, especially in accuracy gain, memory cost, training efficiency, and task-order robustness. Furthermore, to make the non-growing backbone (i.e., a model with limited network capacity) suffice to train on more incoming tasks, a graceful forgetting implementation on previously learned trivial tasks is empirically investigated.

Improve the Performance and Stability of Incremental Learning by a Similarity Harmonizing Mechanism

Disentangling and Mitigating the Impact of Task Similarity for Continual Learning

On Better Exploring and Exploiting Task Relationships in Multitask Learning: Joint Model and Feature Learning.

Modeling Output-Level Task Relatedness in Multi-Task Learning with Feedback Mechanism

Mitigate Negative Transfer with Similarity Heuristic Lifelong Prompt Tuning

Class Incremental Learning with Deep Contrastive Learning and Attention Distillation

Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models

Adversarial Feature Alignment: Avoid Catastrophic Forgetting in Incremental Task Lifelong Learning.

Prototypes Sampling Mechanism for Class Incremental Learning

Learning from Similar Linear Representations: Adaptivity, Minimaxity, and Robustness

Task Similarity Aware Meta Learning - Theory-Inspired Improvement on MAML.

Continual Learning in the Teacher-Student Setup: Impact of Task Similarity

Model Behavior Preserving for Class-Incremental Learning

Imbalance Mitigation for Continual Learning via Knowledge Decoupling and Dual Enhanced Contrastive Learning

Distilled Meta-learning for Multi-Class Incremental Learning

Complementary Learning Subnetworks for Parameter-Efficient Class-Incremental Learning

New Insights on Relieving Task-Recency Bias for Online Class Incremental Learning

A Model-Agnostic Approach to Mitigate Gradient Interference for Multi-Task Learning

Class-Incremental Learning via Knowledge Amalgamation

DualHSIC: HSIC-Bottleneck and Alignment for Continual Learning

Incremental Model Enhancement Via Memory-based Contrastive Learning