Abstract:Continual learning aims to create artificial neural networks capable of accumulating knowledge and skills through incremental training on a sequence of tasks. The main challenge of continual learning is catastrophic interference, wherein new knowledge overrides or interferes with past knowledge, leading to forgetting. An associated issue is the problem of learning "cross-task knowledge," where models fail to acquire and retain knowledge that helps differentiate classes across task boundaries. A common solution to both problems is "replay," where a limited buffer of past instances is utilized to learn cross-task knowledge and mitigate catastrophic interference. However, a notable drawback of these methods is their tendency to overfit the limited replay buffer. In contrast, our proposed solution, SurpriseNet, addresses catastrophic interference by employing a parameter isolation method and learning cross-task knowledge using an auto-encoder inspired by anomaly detection. SurpriseNet is applicable to both structured and unstructured data, as it does not rely on image-specific inductive biases. We have conducted empirical experiments demonstrating the strengths of SurpriseNet on various traditional vision continual-learning benchmarks, as well as on structured data datasets. Source code made available at <a class="link-external link-https" href="https://doi.org/10.5281/zenodo.8247906" rel="external noopener nofollow">this https URL</a> and <a class="link-external link-https" href="https://github.com/tachyonicClock/SurpriseNet-CIKM-23" rel="external noopener nofollow">this https URL</a>

What problem does this paper attempt to address?

The main problems that this paper attempts to solve are catastrophic forgetting and cross - task knowledge acquisition in Class Incremental Learning (Class - IL). Specifically: 1. **Catastrophic Forgetting**: In continuous learning, when a neural network learns a new task, the knowledge of the old task will be overwritten or interfered with by the new knowledge, resulting in a performance decline. This phenomenon is called "catastrophic forgetting". 2. **Cross - task Knowledge Acquisition**: In the Class - Incremental Learning scenario, the model needs to distinguish classes between different tasks, not just the classes within a single task. This is called "cross - task knowledge" or "cross - task class separation". However, existing methods are often difficult to effectively acquire and retain this cross - task knowledge. To solve these problems, the paper proposes **SurpriseNet**, a Class - Incremental Learning method based on anomaly detection. The main features of SurpriseNet include: - **Parameter Isolation**: By pruning unimportant parameters and freezing important ones, a sub - network specific to each task is created, thereby preventing catastrophic forgetting. - **Task Identification**: An auto - encoder is used to infer tasks. By comparing the reconstruction quality of each task - specific sub - network, the task to which the current instance belongs is determined. - **No Replay Mechanism**: Unlike traditional replay methods, SurpriseNet does not rely on storing past training data, avoiding the over - fitting problem of the replay buffer and improving training efficiency. SurpriseNet is suitable for structured and unstructured data and does not rely on image - specific inductive biases, so it has a wider range of applicability. Experimental results show that SurpriseNet performs excellently on low - dimensional data sets, but there are challenges on high - dimensional data sets. For this reason, the paper also proposes a method of using a pre - trained network for dimension reduction (SurpriseNetE) to improve the accuracy of task identification.

Look At Me, No Replay! SurpriseNet: Anomaly Detection Inspired Class Incremental Learning

Generative Feature Replay For Class-Incremental Learning

SynapNet: A Complementary Learning System Inspired Algorithm With Real-Time Application in Multimodal Perception

A Continual Learning Method for Reducing Class Interference Based on Replay

Squeezing More Past Knowledge for Online Class-Incremental Continual Learning

Autonomous Generative Feature Replay for Non-Exemplar Class-Incremental Learning

ONER: Online Experience Replay for Incremental Anomaly Detection

Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes

Continual Recognition with Adaptive Memory Update.

Learning to Continually Learn Rapidly from Few and Noisy Data

Adaptive Memory Replay for Continual Learning

Class-Incremental Learning Using Generative Experience Replay Based on Time-aware Regularization

Not Just Selection, but Exploration: Online Class-Incremental Continual Learning Via Dual View Consistency

Online continual learning through unsupervised mutual information maximization

MixER: Mixup-Based Experience Replay for Online Class-Incremental Learning

May the Forgetting Be with You: Alternate Replay for Learning with Noisy Labels

Class-Incremental Learning Based on Anomaly Detection

Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning

Watch Your Step: Optimal Retrieval for Continual Learning at Scale

Memory Enhanced Replay for Continual Learning

Efficient Non-Exemplar Class-Incremental Learning with Retrospective Feature Synthesis