Abstract:Sentence Representation Learning (SRL) is a fundamental task in Natural Language Processing (NLP), with the Contrastive Learning of Sentence Embeddings (CSE) being the mainstream technique due to its superior performance. An intriguing phenomenon in CSE is the significant performance gap between supervised and unsupervised methods, with their only difference lying in the training data. Previous works attribute this performance gap to differences in two representation properties (alignment and uniformity). However, since alignment and uniformity only measure the results, they fail to answer "What aspects of the training data contribute to the performance gap?" and "How can the performance gap be narrowed?", In this paper, we conduct empirical experiments to answer these "What" and "How" questions. We first answer the "What" question by thoroughly comparing the behavior of supervised and unsupervised CSE during their respective training processes. From the comparison, we identify the similarity pattern as a key factor to the performance gap, and introduce a metric, called Relative Fitting Difficulty (RFD), to measure the complexity of the similarity pattern. Then, based on the insights gained from the "What" question, we tackle the "How" question by increasing the pattern complexity of the training data. We achieve this by leveraging the In-Context Learning (ICL) capability of the Large Language Model (LLM) to generate data that simulates complex patterns. By utilizing the hierarchical patterns in the LLM-generated data, we effectively narrow the gap between supervised and unsupervised CSE. We release our codes and appendix at <a class="link-external link-https" href="https://github.com/BDBC-KG-NLP/NGCSE" rel="external noopener nofollow">this https URL</a>.

CSS-LM: A Contrastive Framework for Semi-Supervised Fine-Tuning of Pre-Trained Language Models

OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding

Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning

Instruction Tuning Vs. In-Context Learning: Revisiting Large Language Models in Few-Shot Computational Social Science

CONCSS: Contrastive-based Context Comprehension for Dialogue-appropriate Prosody in Conversational Speech Synthesis

Multimodal Supervised Contrastive Learning in Remote Sensing Downstream Tasks

Bridging the Gap between Language Models and Cross-Lingual Sequence Labeling

Debiased Contrastive Learning of Unsupervised Sentence Representations

Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learning

Making Pre-trained Language Models End-to-end Few-shot Learners with Contrastive Prompt Tuning

Alleviating Over-smoothing for Unsupervised Sentence Representation.

CoDiM: Learning with Noisy Labels via Contrastive Semi-Supervised Learning

Adversarial Supervised Contrastive Learning

MCLF: A Multi-grained Contrastive Learning Framework for ASR-robust Spoken Language Understanding

Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning

Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling

Learning with noisy labels using collaborative sample selection and contrastive semi-supervised learning

A Multi-level Supervised Contrastive Learning Framework for Low-Resource Natural Language Inference

MuSCLe: A Multi-Strategy Contrastive Learning Framework for Weakly Supervised Semantic Segmentation

Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model