Abstract:Through minimization of an appropriate loss function such as the InfoNCE loss, contrastive learning (CL) learns a useful representation function by pulling positive samples close to each other while pushing negative samples far apart in the embedding space. The positive samples are typically created using "label-preserving" augmentations, i.e., domain-specific transformations of a given datum or anchor. In absence of class information, in unsupervised CL (UCL), the negative samples are typically chosen randomly and independently of the anchor from a preset negative sampling distribution over the entire dataset. This leads to class-collisions in UCL. Supervised CL (SCL), avoids this class collision by conditioning the negative sampling distribution to samples having labels different from that of the anchor. In hard-UCL (H-UCL), which has been shown to be an effective method to further enhance UCL, the negative sampling distribution is conditionally tilted, by means of a hardening function, towards samples that are closer to the anchor. Motivated by this, in this paper we propose hard-SCL (H-SCL) {wherein} the class conditional negative sampling distribution {is tilted} via a hardening function. Our simulation results confirm the utility of H-SCL over SCL with significant performance gains {in downstream classification tasks.} Analytically, we show that {in the} limit of infinite negative samples per anchor and a suitable assumption, the {H-SCL loss} is upper bounded by the {H-UCL loss}, thereby justifying the utility of H-UCL {for controlling} the H-SCL loss in the absence of label information. Through experiments on several datasets, we verify the assumption as well as the claimed inequality between H-UCL and H-SCL losses. We also provide a plausible scenario where H-SCL loss is lower bounded by UCL loss, indicating the limited utility of UCL in controlling the H-SCL loss.

A new contrastive learning framework for reducing the effect of hard negatives

Debiased Graph Contrastive Learning.

ProGCL: Rethinking Hard Negative Mining in Graph Contrastive Learning

Time-Series Contrastive Learning against False Negatives and Class Imbalance

Synthetic Hard Negative Samples for Contrastive Learning

Tion for negative-free symmetric contrastive learning

Progressive Hard Negative Masking: from Global Uniformity to Local Tolerance

Contrastive JS: A Novel Scheme for Enhancing the Accuracy and Robustness of Deep Models

Hard Negative Sampling via Regularized Optimal Transport for Contrastive Representation Learning

Adaptive Negative Representations for Graph Contrastive Learning

Contrastive Learning with Semantic Consistency Constraint

Supervised Contrastive Learning with Hard Negative Samples

Trading Hard Negatives and True Negatives: A Debiased Contrastive Collaborative Filtering Approach.

Contrastive sentence representation learning with adaptive false negative cancellation

A Unified Framework for Contrastive Learning from a Perspective of Affinity Matrix

Contrastive Learning with Synthetic Positives

ContrastNet: A Contrastive Learning Framework for Few-Shot Text Classification

Unsupervised Sentence Representation Via Contrastive Learning with Mixing Negatives

Multiple Contrast Clustering Network

Hard-Negative Sampling for Contrastive Learning: Optimal Representation Geometry and Neural- vs Dimensional-Collapse

HNCSE: Advancing Sentence Embeddings Via Hybrid Contrastive Learning with Hard Negatives