Abstract:Continual learning enables learning systems to adapt to evolving data distributions by sequentially acquiring knowledge from a series of tasks. Unsupervised lifelong learning refers to the ability to learn over time while memorizing previous patterns without supervision. However, the prior methods in this field heavily rely on supervised or reinforcement learning, which necessitates annotated data, thereby limiting their scalability in real-world applications where data is often biased and lacks annotations. To overcome these challenges, this work introduces a novel approach called L ifelong L earning gets better with MixUp and U nsupervised C ontinual R epresentation (LL-UCR). LL-UCR aims to learn feature representations from unlabeled tasks, eliminating the need for annotated data. Within the LL-UCR framework, two innovative techniques are introduced: LL-MixUp, which mitigates catastrophic forgetting by interpolating samples between current and previous tasks, and Dark Experience Replay (DER) Buzzega et al. ( Adv Neural Inf Process Syst , 33 , 15920–15930 2020) adapted for UCR, aligning network logits across tasks. To overcome buffer size limitations in replay-based methods, the Retrospective Adversarial Replay (RAR) framework is incorporated, facilitating diverse replay sample generation. Through systematic analysis, we demonstrate that unsupervised visual representations exhibit remarkable resilience to catastrophic forgetting, consistently outperforming supervised methods in terms of performance and generalization on out-of-distribution tasks. Furthermore, our qualitative analysis reveals that LL-UCR fosters a smoother loss landscape and acquires meaningful feature representations. Extensive experimental evaluations conducted on diverse datasets validate the superior performance of LL-UCR compared to state-of-the-art supervised continual learning methods and the unsupervised LUMP Madaan et al. ( International conference on learning representations , 2020) method, effectively mitigating catastrophic forgetting.

Revisiting Supervision for Continual Representation Learning

Rethinking the Representational Continuity: Towards Unsupervised Continual Learning

Representational Continuity for Unsupervised Continual Learning

Probing Representation Forgetting in Supervised and Unsupervised Continual Learning

Enhancing Visual Continual Learning with Language-Guided Supervision

Beyond Supervised Continual Learning: a Review

R epresentational c ontinuity for u nsupervised c ontinual l earning

Contrastive Supervised Distillation for Continual Representation Learning

Self-supervised Spatiotemporal Representation Learning by Exploiting Video Continuity

Learning Where to Learn in Cross-View Self-Supervised Learning

On the Effectiveness of Supervision in Asymmetric Non-Contrastive Learning

Towards Unsupervised Representation Learning: Learning, Evaluating and Transferring Visual Representations

Understanding and Improving the Role of Projection Head in Self-Supervised Learning

Scaling and Benchmarking Self-Supervised Visual Representation Learning

Deep Supervision with Intermediate Concepts

Lifelong learning gets better with MixUp and unsupervised continual representation

Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Experts

A Survey on Self-Supervised Representation Learning

Online continual learning through unsupervised mutual information maximization

A Closer Look at Rehearsal-Free Continual Learning

Exemplar-free Continual Representation Learning via Learnable Drift Compensation