Abstract:For artificial learning systems, continual learning over time from a stream of data is essential. The burgeoning studies on supervised continual learning have achieved great progress, while the study of catastrophic forgetting in unsupervised learning is still blank. Among unsupervised learning methods, self-supervise learning method shows tremendous potential on visual representation without any labeled data at scale. To improve the visual representation of self-supervised learning, larger and more varied data is needed. In the real world, unlabeled data is generated at all times. This circumstance provides a huge advantage for the learning of the self-supervised method. However, in the current paradigm, packing previous data and current data together and training it again is a waste of time and resources. Thus, a continual self-supervised learning method is badly needed. In this paper, we make the first attempt to implement the continual contrastive self-supervised learning by proposing a rehearsal method, which keeps a few exemplars from the previous data. Instead of directly combining saved exemplars with the current data set for training, we leverage self-supervised knowledge distillation to transfer contrastive information among previous data to the current network by mimicking similarity score distribution inferred by the old network over a set of saved exemplars. Moreover, we build an extra sample queue to assist the network to distinguish between previous and current data and prevent mutual interference while learning their own feature representation. Experimental results show that our method performs well on CIFAR100 and ImageNet-Sub. Compared with self-supervised baselines, which learning tasks one by one without taking any technique, we improve the image classification top-1 accuracy by 1.60% on CIFAR100 and 2.86% on ImageNet-Sub under 10 incremental steps setting.

A soft nearest-neighbor framework for continual semi-supervised learning

A Local Online Learning Approach for Non-linear Data.

Hypernetworks for Continual Semi-Supervised Learning

Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognition

Iterative Nearest Neighborhood Oversampling in Semisupervised Learning from Imbalanced Data

With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations

SemiNLL: A Framework of Noisy-Label Learning by Semi-Supervised Learning

Semi-supervised learning based on nearest neighbor rule and cut edges

Nearest Neighbor-Based Contrastive Learning for Hyperspectral and LiDAR Data Classification

Improving Representational Continuity via Continued Pretraining

Nearest Neighbor Classifier Embedded Network for Active Learning

Enhancing Visual Continual Learning with Language-Guided Supervision

Fairness Continual Learning Approach to Semantic Scene Understanding in Open-World Environments

An Attention-based Representation Distillation Baseline for Multi-Label Continual Learning

Prior-Free Continual Learning with Unlabeled Data in the Wild

MNN: Mixed Nearest-Neighbors for Self-Supervised Learning

Weakly Supervised Instance Segmentation by Exploring Entire Object Regions

Reinforced Continual Learning

Continual Contrastive Self-supervised Learning for Image Classification.

Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learning

Self-supervised Continual Learning for Object Recognition in Image Sequences