Abstract:Contrastive learning (CL) methods effectively learn data representations in a self-supervision manner, where the encoder contrasts each positive sample over multiple negative samples via a one-vs-many softmax cross-entropy loss. By leveraging large amounts of unlabeled image data, recent CL methods have achieved promising results when pretrained on large-scale datasets, such as ImageNet. However, most of them consider the augmented views from the same instance are positive pairs, while views from other instances are negative ones. Such binary partition insufficiently considers the relation between samples and tends to yield worse performance when generalized on images in the wild. In this paper, to further improve the performance of CL and enhance its robustness on various datasets, {we propose a doubly CL strategy that separately compares positive and negative samples within their own groups, and then proceeds with a contrast between positive and negative groups}. We realize this strategy with contrastive attraction and contrastive repulsion (CACR), which makes the query not only exert a greater force to attract more distant positive samples but also do so to repel closer negative samples. Theoretical analysis reveals that CACR generalizes CL's behavior by positive attraction and negative repulsion, and it further considers the intra-contrastive relation within the positive and negative pairs to narrow the gap between the sampled and true distribution, which is important when datasets are less curated. With our extensive experiments, CACR not only demonstrates good performance on CL benchmarks, but also shows better robustness when generalized on imbalanced image datasets. Code and pre-trained checkpoints are available at <a class="link-external link-https" href="https://github.com/JegZheng/CACR-SSL" rel="external noopener nofollow">this https URL</a>.

Consistent Positive Correlation Sample Distribution: Alleviating the Negative Sample Noise Issue in Contrastive Adaptation

Optimal Positive Generation via Latent Transformation for Contrastive Learning

Adversarial Contrastive Estimation

Contrastive Learning with Negative Sampling Correction

Understanding Contrastive Learning via Distributionally Robust Optimization

Mining negative samples on contrastive learning via curricular weighting strategy

Synthetic Hard Negative Samples for Contrastive Learning

Contrastive Attraction and Contrastive Repulsion for Representation Learning

RepCo: Replenish Sample Views with Better Consistency for Contrastive Learning

CaCo: Both Positive and Negative Samples are Directly Learnable via Cooperative-adversarial Contrastive Learning

Ultimate Negative Sampling for Contrastive Learning

Noise is Also Useful: Negative Correlation-Steered Latent Contrastive Learning

Asymmetric Patch Sampling for Contrastive Learning

Negative Samples Mining Matters: Reconsidering Hyperspectral Image Classification with Contrastive Learning

Contrastive JS: A Novel Scheme for Enhancing the Accuracy and Robustness of Deep Models

Boosting Graph Contrastive Learning Via Adaptive Sampling

Improving Contrastive Learning by Visualizing Feature Transformation

Negative Sampling for Contrastive Representation Learning: A Review

Contrastive sentence representation learning with adaptive false negative cancellation

EMC$^2$: Efficient MCMC Negative Sampling for Contrastive Learning with Global Convergence

CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective