Abstract:Learning good representations involves capturing the diverse ways in which data samples relate. Contrastive loss - an objective matching related samples - underlies methods from self-supervised to multimodal learning. Contrastive losses, however, can be viewed more broadly as modifying a similarity graph to indicate how samples should relate in the embedding space. This view reveals a shortcoming in contrastive learning: the similarity graph is binary, as only one sample is the related positive sample. Crucially, similarities \textit{across} samples are ignored. Based on this observation, we revise the standard contrastive loss to explicitly encode how a sample relates to others. We experiment with this new objective, called $\mathbb{X}$-Sample Contrastive, to train vision models based on similarities in class or text caption descriptions. Our study spans three scales: ImageNet-1k with 1 million, CC3M with 3 million, and CC12M with 12 million samples. The representations learned via our objective outperform both contrastive self-supervised and vision-language models trained on the same data across a range of tasks. When training on CC12M, we outperform CLIP by $0.6\%$ on both ImageNet and ImageNet Real. Our objective appears to work particularly well in lower-data regimes, with gains over CLIP of $16.8\%$ on ImageNet and $18.1\%$ on ImageNet Real when training with CC3M. Finally, our objective seems to encourage the model to learn representations that separate objects from their attributes and backgrounds, with gains of $3.3$-$5.6$\% over CLIP on ImageNet9. We hope the proposed solution takes a small step towards developing richer learning objectives for understanding sample relations in foundation models.

SimO Loss: Anchor-Free Contrastive Loss for Fine-Grained Supervised Contrastive Learning

An Asymmetric Contrastive Loss for Handling Imbalanced Datasets

Identical and Fraternal Twins: Fine-Grained Semantic Contrastive Learning of Sentence Representations

Contrastive Learning Is Spectral Clustering On Similarity Graph

$\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs

Multi-Similarity Contrastive Learning

Improving Multi-lingual Alignment Through Soft Contrastive Learning

Adaptive Multi-head Contrastive Learning

Inter-Class Angular Loss for Convolutional Neural Networks.

Decoupled Contrastive Learning for Long-Tailed Recognition

Similarity-Dissimilarity Loss with Supervised Contrastive Learning for Multi-label Classification

On the Importance of Contrastive Loss in Multimodal Learning

ACF: Aligned Contrastive Finetuning For Language and Vision Tasks

Weighted Point Cloud Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metric

A Unified Framework for Contrastive Learning from a Perspective of Affinity Matrix

Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP)

Triangular Contrastive Learning on Molecular Graphs

Distortion-Disentangled Contrastive Learning.

Neighborhood-Enhanced Supervised Contrastive Learning for Collaborative Filtering

Contrastive sentence representation learning with adaptive false negative cancellation

Learning the Relation between Similarity Loss and Clustering Loss in Self-Supervised Learning