Abstract:Self-supervised learning (SSL) is a machine learning approach where the data itself provides supervision, eliminating the need for external labels. The model is forced to learn about the data structure or context by solving a pretext task. With SSL, models can learn from abundant and cheap unlabeled data, significantly reducing the cost of training models where labels are expensive or inaccessible. In Computer Vision, SSL is widely used as pre-training followed by a downstream task, such as supervised transfer, few-shot learning on smaller labeled data sets, and/or unsupervised clustering. Unfortunately, it is infeasible to evaluate SSL methods on all possible downstream tasks and objectively measure the quality of the learned representation. Instead, SSL methods are evaluated using in-domain evaluation protocols, such as fine-tuning, linear probing, and k-nearest neighbors (kNN). However, it is not well understood how well these evaluation protocols estimate the representation quality of a pre-trained model for different downstream tasks under different conditions, such as dataset, metric, and model architecture. We study how classification-based evaluation protocols for SSL correlate and how well they predict downstream performance on different dataset types. Our study includes eleven common image datasets and 26 models that were pre-trained with different SSL methods or have different model backbones. We find that in-domain linear/kNN probing protocols are, on average, the best general predictors for out-of-domain performance. We further investigate the importance of batch normalization and evaluate how robust correlations are for different kinds of dataset domain shifts. We challenge assumptions about the relationship between discriminative and generative self-supervised methods, finding that most of their performance differences can be explained by changes to model backbones.

On Pretraining Data Diversity for Self-Supervised Learning

Using Self-supervised Learning Can Improve Model Fairness

On the Connection between Pre-training Data Diversity and Fine-tuning Robustness

Bridging Diversity and Uncertainty in Active learning with Self-Supervised Pre-Training

Feature diversity in self-supervised learning

Benchmarking Self-Supervised Learning on Diverse Pathology Datasets

Augmentations vs Algorithms: What Works in Self-Supervised Learning

On the Trade-off of Intra-/Inter-class Diversity for Supervised Pre-training

Exploring the Effect of Dataset Diversity in Self-Supervised Learning for Surgical Computer Vision

On Diversity in Discriminative Neural Networks

Evaluating Fairness in Self-supervised and Supervised Models for Sequential Data

Self-supervised Learning is More Robust to Dataset Imbalance

On Improving the Algorithm-, Model-, and Data- Efficiency of Self-Supervised Learning

Dive into Self-Supervised Learning for Medical Image Analysis: Data, Models and Tasks

A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classification

Dive into the Details of Self-Supervised Learning for Medical Image Analysis.

When Semi-Supervised Learning Meets Transfer Learning: Training Strategies, Models and Datasets.

Self-supervised visual learning in the low-data regime: a comparative evaluation

A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends

Diversity in Machine Learning

Semi-supervised Learning Regularized by Adversarial Perturbation and Diversity Maximization.