Abstract:Self-Supervised Learning (SSL) is a valuable and robust training methodology for contemporary Deep Neural Networks (DNNs), enabling unsupervised pretraining on a `pretext task' that does not require ground-truth labels/annotation. This allows efficient representation learning from massive amounts of unlabeled training data, which in turn leads to increased accuracy in a `downstream task' by exploiting supervised transfer learning. Despite the relatively straightforward conceptualization and applicability of SSL, it is not always feasible to collect and/or to utilize very large pretraining datasets, especially when it comes to real-world application settings. In particular, in cases of specialized and domain-specific application scenarios, it may not be achievable or practical to assemble a relevant image pretraining dataset in the order of millions of instances or it could be computationally infeasible to pretrain at this scale. This motivates an investigation on the effectiveness of common SSL pretext tasks, when the pretraining dataset is of relatively limited/constrained size. In this context, this work introduces a taxonomy of modern visual SSL methods, accompanied by detailed explanations and insights regarding the main categories of approaches, and, subsequently, conducts a thorough comparative experimental evaluation in the low-data regime, targeting to identify: a) what is learnt via low-data SSL pretraining, and b) how do different SSL categories behave in such training scenarios. Interestingly, for domain-specific downstream tasks, in-domain low-data SSL pretraining outperforms the common approach of large-scale pretraining on general datasets. Grounded on the obtained results, valuable insights are highlighted regarding the performance of each category of SSL methods, which in turn suggest straightforward future research directions in the field.

Progress and Thinking on Self-Supervised Learning Methods in Computer Vision: A Review

A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends

A Survey of Self-Supervised Learning from Multiple Perspectives: Algorithms, Theory, Applications and Future Trends

Self-supervised Learning on Graphs: Contrastive, Generative,or Predictive

Self-Supervised Learning in Remote Sensing: A review

Self-Supervised Learning and Its Applications in Ophthalmology

Semi-Supervised and Unsupervised Deep Visual Learning: A Survey

Self-supervised Learning: A Succinct Review

Consequential Advancements of Self-Supervised Learning (SSL) in Deep Learning Contexts

Self-supervised learning for medical image analysis: a comprehensive review

Review on self-supervised image recognition using deep neural networks

Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey

A review on discriminative self-supervised learning methods

A Review of Predictive and Contrastive Self-supervised Learning for Medical Images

A Short Review on Supervised Machine Learning and Deep Learning Techniques in Computer Vision

Deep learning in computer vision: A critical review of emerging techniques and application scenarios

Dive into the Details of Self-Supervised Learning for Medical Image Analysis.

Connecting the Dots in Self-Supervised Learning: A Brief Survey for Beginners

Self-supervised visual learning in the low-data regime: a comparative evaluation

A Survey of the Self Supervised Learning Mechanisms for Vision Transformers

Computer Vision Self-supervised Learning Methods on Time Series