Abstract:For artificial learning systems, continual learning over time from a stream of data is essential. The burgeoning studies on supervised continual learning have achieved great progress, while the study of catastrophic forgetting in unsupervised learning is still blank. Among unsupervised learning methods, self-supervise learning method shows tremendous potential on visual representation without any labeled data at scale. To improve the visual representation of self-supervised learning, larger and more varied data is needed. In the real world, unlabeled data is generated at all times. This circumstance provides a huge advantage for the learning of the self-supervised method. However, in the current paradigm, packing previous data and current data together and training it again is a waste of time and resources. Thus, a continual self-supervised learning method is badly needed. In this paper, we make the first attempt to implement the continual contrastive self-supervised learning by proposing a rehearsal method, which keeps a few exemplars from the previous data. Instead of directly combining saved exemplars with the current data set for training, we leverage self-supervised knowledge distillation to transfer contrastive information among previous data to the current network by mimicking similarity score distribution inferred by the old network over a set of saved exemplars. Moreover, we build an extra sample queue to assist the network to distinguish between previous and current data and prevent mutual interference while learning their own feature representation. Experimental results show that our method performs well on CIFAR100 and ImageNet-Sub. Compared with self-supervised baselines, which learning tasks one by one without taking any technique, we improve the image classification top-1 accuracy by 1.60% on CIFAR100 and 2.86% on ImageNet-Sub under 10 incremental steps setting.

A Novel Self-Learning Network Integrating Contrastive Learning, Perceptual Learning and Masked Image Modelling

A Novel Contrastive Learning Model for Aerial Images

Masking Improves Contrastive Self-Supervised Learning for ConvNets, and Saliency Tells You Where

Saliency Guided Contrastive Learning on Scene Images

SiSL-Net: Saliency-guided self-supervised learning network for image classification

Masked Contrastive Representation Learning

ContrastNet: Unsupervised Feature Learning by Autoencoder and Prototypical Contrastive Learning for Hyperspectral Imagery Classification.

A self-supervised image aesthetic assessment combining masked image modeling and contrastive learning

Self-Supervised Visual Representations Learning by Contrastive Mask Prediction

Continual Contrastive Self-supervised Learning for Image Classification.

Learning with Unmasked Tokens Drives Stronger Vision Learners

ADCL: Adversarial Distilled Contrastive Learning on lightweight models for self-supervised image classification

Self-Supervised Intra-Modal and Cross-Modal Contrastive Learning for Point Cloud Understanding

Masked Image Modeling with Denoising Contrast

A simple, efficient and scalable contrastive masked autoencoder for learning visual representations

Self-Supervised Self-Organizing Clustering Network: A Novel Unsupervised Representation Learning Method

Contrastive Masked Autoencoders are Stronger Vision Learners

Masked Modeling for Self-supervised Representation Learning on Vision and Beyond

MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining

Complementary Mask Self-Supervised Pre-training Based on Teacher-Student Network

PCMask: A Dual-Branch Self-supervised Medical Image Segmentation Method Using Pixel-Level Contrastive Learning and Masked Image Modeling.