Abstract:Sentence Representation Learning (SRL) is a fundamental task in Natural Language Processing (NLP), with the Contrastive Learning of Sentence Embeddings (CSE) being the mainstream technique due to its superior performance. An intriguing phenomenon in CSE is the significant performance gap between supervised and unsupervised methods, with their only difference lying in the training data. Previous works attribute this performance gap to differences in two representation properties (alignment and uniformity). However, since alignment and uniformity only measure the results, they fail to answer "What aspects of the training data contribute to the performance gap?" and "How can the performance gap be narrowed?", In this paper, we conduct empirical experiments to answer these "What" and "How" questions. We first answer the "What" question by thoroughly comparing the behavior of supervised and unsupervised CSE during their respective training processes. From the comparison, we identify the similarity pattern as a key factor to the performance gap, and introduce a metric, called Relative Fitting Difficulty (RFD), to measure the complexity of the similarity pattern. Then, based on the insights gained from the "What" question, we tackle the "How" question by increasing the pattern complexity of the training data. We achieve this by leveraging the In-Context Learning (ICL) capability of the Large Language Model (LLM) to generate data that simulates complex patterns. By utilizing the hierarchical patterns in the LLM-generated data, we effectively narrow the gap between supervised and unsupervised CSE. We release our codes and appendix at <a class="link-external link-https" href="https://github.com/BDBC-KG-NLP/NGCSE" rel="external noopener nofollow">this https URL</a>.

CCDC: A Chinese-Centric Cross Domain Contrastive Learning Framework

OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding

A Contrastive Framework to Enhance Unsupervised Sentence Representation Learning

SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives

Enhancing Multilingual Universal Sentence Embeddings by Monolingual Contrastive Learning

HNCSE: Advancing Sentence Embeddings Via Hybrid Contrastive Learning with Hard Negatives

DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings

Unsupervised Sentence Embedding Model Based on Contrastive Learning

SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samples

Semantic-Enhanced Supervised Contrastive Learning.

InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings

Dual Contrastive Learning for Cross-domain Named Entity Recognition

Improving Contrastive Learning of Sentence Embeddings with Focal-InfoNCE

HCSC: Hierarchical Contrastive Selective Coding

Mere Contrastive Learning for Cross-Domain Sentiment Analysis

DebCSE: Rethinking Unsupervised Contrastive Sentence Embedding Learning in the Debiasing Perspective

Instance Smoothed Contrastive Learning for Unsupervised Sentence Embedding

C3T: Contrastive Consistency Cross-Network Learning for Semi-Supervised Semantic Segmentation

JCSE: Contrastive Learning of Japanese Sentence Embeddings and Its Applications

Contrastive sentence representation learning with adaptive false negative cancellation

Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model