Abstract:Learning good representations involves capturing the diverse ways in which data samples relate. Contrastive loss - an objective matching related samples - underlies methods from self-supervised to multimodal learning. Contrastive losses, however, can be viewed more broadly as modifying a similarity graph to indicate how samples should relate in the embedding space. This view reveals a shortcoming in contrastive learning: the similarity graph is binary, as only one sample is the related positive sample. Crucially, similarities \textit{across} samples are ignored. Based on this observation, we revise the standard contrastive loss to explicitly encode how a sample relates to others. We experiment with this new objective, called $\mathbb{X}$-Sample Contrastive, to train vision models based on similarities in class or text caption descriptions. Our study spans three scales: ImageNet-1k with 1 million, CC3M with 3 million, and CC12M with 12 million samples. The representations learned via our objective outperform both contrastive self-supervised and vision-language models trained on the same data across a range of tasks. When training on CC12M, we outperform CLIP by $0.6\%$ on both ImageNet and ImageNet Real. Our objective appears to work particularly well in lower-data regimes, with gains over CLIP of $16.8\%$ on ImageNet and $18.1\%$ on ImageNet Real when training with CC3M. Finally, our objective seems to encourage the model to learn representations that separate objects from their attributes and backgrounds, with gains of $3.3$-$5.6$\% over CLIP on ImageNet9. We hope the proposed solution takes a small step towards developing richer learning objectives for understanding sample relations in foundation models.

The Impact of Negative Sampling on Contrastive Structured World Models

Debiased Graph Contrastive Learning.

OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding

When hard negative sampling meets supervised contrastive learning

Event Representation Via Contrastive Learning with Prototype Based Hard Negative Sampling

Contrasting the landscape of contrastive and non-contrastive learning

Negative Sampling for Contrastive Representation Learning: A Review

Adversarial Contrastive Estimation

Contrastive Learning of Structured World Models

Enhancing Conceptual Understanding in Multimodal Contrastive Learning through Hard Negative Samples

ProSampler: Improving Contrastive Learning by Better Mini-batch Sampling

Analysis of the Impact of Negative Sampling on Link Prediction in Knowledge Graphs

Mining negative samples on contrastive learning via curricular weighting strategy

Improving Negative Sampling for Word Representation Using Self-embedded Features

AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries

Learning Determinantal Point Processes by Corrective Negative Sampling

Hard Negative Sampling via Regularized Optimal Transport for Contrastive Representation Learning

$\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs

Global Contrastive Batch Sampling via Optimization on Sample Permutations

Contrastive Unsupervised Learning of World Model with Invariant Causal Features

Boosting Graph Contrastive Learning Via Adaptive Sampling