Abstract:Unsupervised feature learning has made great strides with contrastive learning based on instance discrimination and invariant mapping, as benchmarked on curated class-balanced datasets. However, natural data could be highly correlated and long-tail distributed. Natural between-instance similarity conflicts with the presumed instance distinction, causing unstable training and poor performance. Our idea is to discover and integrate between-instance similarity into contrastive learning, not directly by instance grouping, but by cross-level discrimination (CLD) between instances and local instance groups. While invariant mapping of each instance is imposed by attraction within its augmented views, between-instance similarity could emerge from common repulsion against instance groups. Our batch-wise and cross-view comparisons also greatly improve the positive/negative sample ratio of contrastive learning and achieve better invariant mapping. To effect both grouping and discrimination objectives, we impose them on features separately derived from a shared representation. In addition, we propose normalized projection heads and unsupervised hyper-parameter tuning for the first time. Our extensive experimentation demonstrates that CLD is a lean and powerful add-on to existing methods such as NPID, MoCo, InfoMin, and BYOL on highly correlated, long-tail, or balanced datasets. It not only achieves new state-of-the-art on self-supervision, semi-supervision, and transfer learning benchmarks, but also beats MoCo v2 and SimCLR on every reported performance attained with a much larger compute. CLD effectively brings unsupervised learning closer to natural data and real-world applications. Our code is publicly available at: <a class="link-external link-https" href="https://github.com/frank-xwang/CLD-UnsupervisedLearning" rel="external noopener nofollow">this https URL</a>.

Fine-Grained Representation Learning via Multi-Level Contrastive Learning without Class Priors

Multi-Grained Contrast for Data-Efficient Unsupervised Representation Learning

Towards a Unified Framework of Contrastive Learning for Disentangled Representations

Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Framework

Multi-level Disentangled Contrastive Learning on Heterogeneous Graphs

Unsupervised Graph-Level Representation Learning with Hierarchical Contrasts

Contrastively Enforcing Distinctiveness for Multi-Label Image Classification

Learning Disentangled Representation by Exploiting Pretrained Generative Models: A Contrastive Learning View

Distortion-Disentangled Contrastive Learning.

Hyperbolic Contrastive Learning for Visual Representations beyond Objects

HAPiCLR: heuristic attention pixel-level contrastive loss representation learning for self-supervised pretraining

Multi-label Cluster Discrimination for Visual Representation Learning

Debiased Contrastive Learning of Unsupervised Sentence Representations

Representation Learning with Fine-grained Patterns

Prototypical Contrastive Learning of Unsupervised Representations

Distillation with Contrast is All You Need for Self-Supervised Point Cloud Representation Learning

Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination

Hierarchical contrastive representation for zero shot learning

Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognition

CrCD: Multi-Direction-MLP-based Cross-Contrastive Disambiguation for Hyperspectral Image Partial Label Learning

Deep Contrastive Representation Learning with Self-Distillation