Abstract:Unsupervised contrastive learning has achieved outstanding success, while the mechanism of contrastive loss has been less studied. In this paper, we concentrate on the understanding of the behaviours of unsupervised contrastive loss. We will show that the contrastive loss is a hardness-aware loss function, and the temperature {\tau} controls the strength of penalties on hard negative samples. The previous study has shown that uniformity is a key property of contrastive learning. We build relations between the uniformity and the temperature {\tau} . We will show that uniformity helps the contrastive learning to learn separable features, however excessive pursuit to the uniformity makes the contrastive loss not tolerant to semantically similar samples, which may break the underlying semantic structure and be harmful to the formation of features useful for downstream tasks. This is caused by the inherent defect of the instance discrimination objective. Specifically, instance discrimination objective tries to push all different instances apart, ignoring the underlying relations between samples. Pushing semantically consistent samples apart has no positive effect for acquiring a prior informative to general downstream tasks. A well-designed contrastive loss should have some extents of tolerance to the closeness of semantically similar samples. Therefore, we find that the contrastive loss meets a uniformity-tolerance dilemma, and a good choice of temperature can compromise these two properties properly to both learn separable features and tolerant to semantically similar samples, improving the feature qualities and the downstream performances.

Contrast Is All You Need

Evaluating Large Language Models Using Contrast Sets: An Experimental Approach

Contrastive estimation reveals topic posterior information to linear models

Contrasting the landscape of contrastive and non-contrastive learning

Contrastive Instruction Tuning

Contrastive Learning for Legal Judgment Prediction

Contrastive Quant: Quantization Makes Stronger Contrastive Learning

MetAug: Contrastive Learning via Meta Feature Augmentation

Exploiting Contrastive Learning and Numerical Evidence for Confusing Legal Judgment Prediction

On The Inadequacy of Optimizing Alignment and Uniformity in Contrastive Learning of Sentence Representations

From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data

Contrastive Quant

Achieving Stronger Generation Via Simple Contrastive Tuning

Learning Dynamics of LLM Finetuning

Understanding the Behaviour of Contrastive Loss

The Power of Contrast for Feature Learning: A Theoretical Analysis

Let the CAT out of the bag: Contrastive Attributed explanations for Text

Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning

CLAMP: Contrastive LAnguage Model Prompt-tuning

Understanding Contrastive Learning Through the Lens of Margins

CSS-LM: A Contrastive Framework for Semi-Supervised Fine-Tuning of Pre-Trained Language Models