Abstract:Sentence Representation Learning (SRL) is a fundamental task in Natural Language Processing (NLP), with the Contrastive Learning of Sentence Embeddings (CSE) being the mainstream technique due to its superior performance. An intriguing phenomenon in CSE is the significant performance gap between supervised and unsupervised methods, with their only difference lying in the training data. Previous works attribute this performance gap to differences in two representation properties (alignment and uniformity). However, since alignment and uniformity only measure the results, they fail to answer "What aspects of the training data contribute to the performance gap?" and "How can the performance gap be narrowed?", In this paper, we conduct empirical experiments to answer these "What" and "How" questions. We first answer the "What" question by thoroughly comparing the behavior of supervised and unsupervised CSE during their respective training processes. From the comparison, we identify the similarity pattern as a key factor to the performance gap, and introduce a metric, called Relative Fitting Difficulty (RFD), to measure the complexity of the similarity pattern. Then, based on the insights gained from the "What" question, we tackle the "How" question by increasing the pattern complexity of the training data. We achieve this by leveraging the In-Context Learning (ICL) capability of the Large Language Model (LLM) to generate data that simulates complex patterns. By utilizing the hierarchical patterns in the LLM-generated data, we effectively narrow the gap between supervised and unsupervised CSE. We release our codes and appendix at <a class="link-external link-https" href="https://github.com/BDBC-KG-NLP/NGCSE" rel="external noopener nofollow">this https URL</a>.

Sentence-aware Contrastive Learning for Open-Domain Passage Retrieval

OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding

SynC: A Dense Retrieval Method based on Syntactical Contrastive Learning.

Contrastive Refinement for Dense Retrieval Inference in the Open-Domain Question Answering Task

Simple Flow-Based Contrastive Learning for BERT Sentence Representations

Contrastive sentence representation learning with adaptive false negative cancellation

Debiased Contrastive Learning of Unsupervised Sentence Representations

Contrastive Learning Models for Sentence Representations

PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval.

Span prompt dense passage retrieval for Chinese open domain question answering

Query-as-context Pre-training for Dense Passage Retrieval

Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training

Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models

Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representations

SparseCL: Sparse Contrastive Learning for Contradiction Retrieval

Sentence salience contrastive learning for abstractive text summarization

CAVL: Learning Contrastive and Adaptive Representations of Vision and Language

Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model

Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings

M3: A Multi-Task Mixed-Objective Learning Framework for Open-Domain Multi-Hop Dense Sentence Retrieval

Towards Better Understanding of Contrastive Sentence Representation Learning: A Unified Paradigm for Gradient