Abstract:Self-supervised learning (SSL) is a machine learning approach where the data itself provides supervision, eliminating the need for external labels. The model is forced to learn about the data structure or context by solving a pretext task. With SSL, models can learn from abundant and cheap unlabeled data, significantly reducing the cost of training models where labels are expensive or inaccessible. In Computer Vision, SSL is widely used as pre-training followed by a downstream task, such as supervised transfer, few-shot learning on smaller labeled data sets, and/or unsupervised clustering. Unfortunately, it is infeasible to evaluate SSL methods on all possible downstream tasks and objectively measure the quality of the learned representation. Instead, SSL methods are evaluated using in-domain evaluation protocols, such as fine-tuning, linear probing, and k-nearest neighbors (kNN). However, it is not well understood how well these evaluation protocols estimate the representation quality of a pre-trained model for different downstream tasks under different conditions, such as dataset, metric, and model architecture. We study how classification-based evaluation protocols for SSL correlate and how well they predict downstream performance on different dataset types. Our study includes eleven common image datasets and 26 models that were pre-trained with different SSL methods or have different model backbones. We find that in-domain linear/kNN probing protocols are, on average, the best general predictors for out-of-domain performance. We further investigate the importance of batch normalization and evaluate how robust correlations are for different kinds of dataset domain shifts. We challenge assumptions about the relationship between discriminative and generative self-supervised methods, finding that most of their performance differences can be explained by changes to model backbones.

Benchmarking Self-Supervised Learning for Single-Cell Data

Delineating the Effective Use of Self-Supervised Learning in Single-Cell Genomics

Benchmarking Self-Supervised Learning on Diverse Pathology Datasets

Evaluating Self-Supervised Learning in Medical Imaging: A Benchmark for Robustness, Generalizability, and Multi-Domain Impact

Self-supervision advances morphological profiling by unlocking powerful image representations

A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classification

Learning Where to Learn in Cross-View Self-Supervised Learning

Self-Supervised Representation Learning for Basecalling Nanopore Sequencing Data

CCRL: Contrastive Cell Representation Learning

A Self-Supervised Learning Approach for High Throughput and High Content Cell Segmentation

Dissecting self-supervised learning methods for surgical computer vision

The impacts of active and self-supervised learning on efficient annotation of single-cell expression data

Dive into Self-Supervised Learning for Medical Image Analysis: Data, Models and Tasks

Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?

Label-free Monitoring of Self-Supervised Learning Progress

Self-Supervised Anomaly Detection in the Wild: Favor Joint Embeddings Methods

A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends

Dive into the Details of Self-Supervised Learning for Medical Image Analysis.

Rethinking Self-Supervised Learning: Small is Beautiful

Augmentations vs Algorithms: What Works in Self-Supervised Learning

Single-branch Self-Supervised Learning with Hybrid Tasks