Abstract:Continual Learning research typically focuses on tackling the phenomenon of catastrophic forgetting in neural networks. Catastrophic forgetting is associated with an abrupt loss of knowledge previously learned by a model when the task, or more broadly the data distribution, being trained on changes. In supervised learning problems this forgetting, resulting from a change in the model's representation, is typically measured or observed by evaluating the decrease in old task performance. However, a model's representation can change without losing knowledge about prior tasks. In this work we consider the concept of representation forgetting, observed by using the difference in performance of an optimal linear classifier before and after a new task is introduced. Using this tool we revisit a number of standard continual learning benchmarks and observe that, through this lens, model representations trained without any explicit control for forgetting often experience small representation forgetting and can sometimes be comparable to methods which explicitly control for forgetting, especially in longer task sequences. We also show that representation forgetting can lead to new insights on the effect of model capacity and loss function used in continual learning. Based on our results, we show that a simple yet competitive approach is to learn representations continually with standard supervised contrastive learning while constructing prototypes of class samples when queried on old samples.

Probing Representation Forgetting in Supervised and Unsupervised Continual Learning

Progressive Learning without Forgetting

Feature Forgetting in Continual Representation Learning

Representation Space Maintenance: Against Forgetting in Continual Learning

Rethinking the Representational Continuity: Towards Unsupervised Continual Learning

Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping

Representational Continuity for Unsupervised Continual Learning

Controlling Forgetting with Test-Time Data in Continual Learning

R epresentational c ontinuity for u nsupervised c ontinual l earning

Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgetting

Continual Learning by Modeling Intra-Class Variation

Understanding Forgetting in Continual Learning with Linear Regression

Adaptive Progressive Continual Learning.

Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics

Continual Recognition with Adaptive Memory Update.

Revisiting Supervision for Continual Representation Learning

Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting

An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning

Meta-Learning Representations for Continual Learning

Quantum Continual Learning Overcoming Catastrophic Forgetting

Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learning