Abstract:Continual learning (CL) aims to incrementally learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. Most CL works focus on tackling catastrophic forgetting under a learning-from-scratch paradigm. However, with the increasing prominence of foundation models, pre-trained models equipped with informative representations have become available for various downstream requirements. Several CL methods based on pre-trained models have been explored, either utilizing pre-extracted features directly (which makes bridging distribution gaps challenging) or incorporating adaptors (which may be subject to forgetting). In this paper, we propose a concise and effective approach for CL with pre-trained models. Given that forgetting occurs during parameter updating, we contemplate an alternative approach that exploits training-free random projectors and class-prototype accumulation, which thus bypasses the issue. Specifically, we inject a frozen Random Projection layer with nonlinear activation between the pre-trained model's feature representations and output head, which captures interactions between features with expanded dimensionality, providing enhanced linear separability for class-prototype-based CL. We also demonstrate the importance of decorrelating the class-prototypes to reduce the distribution disparity when using pre-trained representations. These techniques prove to be effective and circumvent the problem of forgetting for both class- and domain-incremental continual learning. Compared to previous methods applied to pre-trained ViT-B/16 models, we reduce final error rates by between 20% and 62% on seven class-incremental benchmarks, despite not using any rehearsal memory. We conclude that the full potential of pre-trained models for simple, effective, and fast CL has not hitherto been fully tapped. Code is at <a class="link-external link-http" href="http://github.com/RanPAC/RanPAC" rel="external noopener nofollow">this http URL</a>.

Premonition: Using Generative Models to Preempt Future Data Changes in Continual Learning

Learn, Imagine and Create: Text-to-Image Generation from Prior Knowledge.

Continual Pre-Training Mitigates Forgetting in Language and Vision

Continual Learning with Pretrained Backbones by Tuning in the Input Space

Analysis of Continual Learning Techniques for Image Generative Models with Learned Class Information Management

Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora

Making the Most of What You Have: Adapting Pre-trained Visual Language Models in the Low-data Regime

On the Usage of Continual Learning for Out-of-Distribution Generalization in Pre-trained Language Models of Code

GPT4Image: Can Large Pre-trained Models Help Vision Models on Perception Tasks?

Investigating Continual Pretraining in Large Language Models: Insights and Implications

Improving Language Understanding by Generative Pre-Training

Continual Learning of Image Classes with Language Guidance from a Vision-Language Model

Continual Learning with Pre-Trained Models: A Survey

CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learning

RanPAC: Random Projections and Pre-trained Models for Continual Learning

How Useful is Continued Pre-Training for Generative Unsupervised Domain Adaptation?

Lifelong generative modeling

Adaptive Progressive Continual Learning.

Generative Continual Concept Learning

Class Incremental Learning with Pre-trained Vision-Language Models