Abstract:For artificial learning systems, continual learning over time from a stream of data is essential. The burgeoning studies on supervised continual learning have achieved great progress, while the study of catastrophic forgetting in unsupervised learning is still blank. Among unsupervised learning methods, self-supervise learning method shows tremendous potential on visual representation without any labeled data at scale. To improve the visual representation of self-supervised learning, larger and more varied data is needed. In the real world, unlabeled data is generated at all times. This circumstance provides a huge advantage for the learning of the self-supervised method. However, in the current paradigm, packing previous data and current data together and training it again is a waste of time and resources. Thus, a continual self-supervised learning method is badly needed. In this paper, we make the first attempt to implement the continual contrastive self-supervised learning by proposing a rehearsal method, which keeps a few exemplars from the previous data. Instead of directly combining saved exemplars with the current data set for training, we leverage self-supervised knowledge distillation to transfer contrastive information among previous data to the current network by mimicking similarity score distribution inferred by the old network over a set of saved exemplars. Moreover, we build an extra sample queue to assist the network to distinguish between previous and current data and prevent mutual interference while learning their own feature representation. Experimental results show that our method performs well on CIFAR100 and ImageNet-Sub. Compared with self-supervised baselines, which learning tasks one by one without taking any technique, we improve the image classification top-1 accuracy by 1.60% on CIFAR100 and 2.86% on ImageNet-Sub under 10 incremental steps setting.

Modeling Inner- and Cross-Task Contrastive Relations for Continual Image Classification

Mitigating Catastrophic Forgetting in Task-Incremental Continual Learning with Adaptive Classification Criterion

Online Continual Learning with Contrastive Vision Transformer

Generalized Few-Shot Continual Learning with Contrastive Mixture of Adapters

Continual Contrastive Self-supervised Learning for Image Classification.

Continual Contrastive Learning for Image Classification

Continual Vision-Language Representation Learning with Off-Diagonal Information

Cross-Modal Alternating Learning with Task-Aware Representations for Continual Learning

Continual Contrastive Learning for Cross-Dataset Scene Classification

Adaptive Multi-head Contrastive Learning

Provable Contrastive Continual Learning

Contrastive Attraction and Contrastive Repulsion for Representation Learning

RegionCL: Can Simple Region Swapping Contribute to Contrastive Learning?

Don't Stop Learning: Towards Continual Learning for the CLIP Model

Look-Ahead Selective Plasticity for Continual Learning of Visual Tasks

Bi-Objective Continual Learning: Learning 'new' While Consolidating 'Known'

Iclip: Bridging Image Classification and Contrastive Language-Image Pre-Training for Visual Recognition

Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning

Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Images

Imbalance Mitigation for Continual Learning via Knowledge Decoupling and Dual Enhanced Contrastive Learning

Triplet Contrastive Learning Framework With Adversarial Hard-Negative Sample Generation for Multimodal Remote Sensing Images