Abstract:Learning good representations involves capturing the diverse ways in which data samples relate. Contrastive loss - an objective matching related samples - underlies methods from self-supervised to multimodal learning. Contrastive losses, however, can be viewed more broadly as modifying a similarity graph to indicate how samples should relate in the embedding space. This view reveals a shortcoming in contrastive learning: the similarity graph is binary, as only one sample is the related positive sample. Crucially, similarities \textit{across} samples are ignored. Based on this observation, we revise the standard contrastive loss to explicitly encode how a sample relates to others. We experiment with this new objective, called $\mathbb{X}$-Sample Contrastive, to train vision models based on similarities in class or text caption descriptions. Our study spans three scales: ImageNet-1k with 1 million, CC3M with 3 million, and CC12M with 12 million samples. The representations learned via our objective outperform both contrastive self-supervised and vision-language models trained on the same data across a range of tasks. When training on CC12M, we outperform CLIP by $0.6\%$ on both ImageNet and ImageNet Real. Our objective appears to work particularly well in lower-data regimes, with gains over CLIP of $16.8\%$ on ImageNet and $18.1\%$ on ImageNet Real when training with CC3M. Finally, our objective seems to encourage the model to learn representations that separate objects from their attributes and backgrounds, with gains of $3.3$-$5.6$\% over CLIP on ImageNet9. We hope the proposed solution takes a small step towards developing richer learning objectives for understanding sample relations in foundation models.

Contrastive Learning Is Spectral Clustering On Similarity Graph

Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysis

Understanding Contrastive Learning via Gaussian Mixture Models

How does Contrastive Learning Organize Images?

$\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs

InfoGCL: Information-Aware Graph Contrastive Learning

Localized Contrastive Learning on Graphs

Understanding and Generalizing Contrastive Learning from the Inverse Optimal Transport Perspective.

Learning the Relation between Similarity Loss and Clustering Loss in Self-Supervised Learning

Localized Graph Contrastive Learning

Graph Clustering with High-Order Contrastive Learning

Dual Contrastive Learning Network for Graph Clustering

Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE

Graph Contrastive Learning with Stable and Scalable Spectral Encoding

S3GCL: Spectral, Swift, Spatial Graph Contrastive Learning

Multi-Graph Contrastive Learning Clustering Network

A self-supervised learning of semantic feature consistency for image clustering

Adversarial Contrastive Learning via Asymmetric InfoNCE.

On the Importance of Contrastive Loss in Multimodal Learning

Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss

Spectral Temporal Contrastive Learning