Abstract:Sentence Representation Learning (SRL) is a fundamental task in Natural Language Processing (NLP), with the Contrastive Learning of Sentence Embeddings (CSE) being the mainstream technique due to its superior performance. An intriguing phenomenon in CSE is the significant performance gap between supervised and unsupervised methods, with their only difference lying in the training data. Previous works attribute this performance gap to differences in two representation properties (alignment and uniformity). However, since alignment and uniformity only measure the results, they fail to answer "What aspects of the training data contribute to the performance gap?" and "How can the performance gap be narrowed?", In this paper, we conduct empirical experiments to answer these "What" and "How" questions. We first answer the "What" question by thoroughly comparing the behavior of supervised and unsupervised CSE during their respective training processes. From the comparison, we identify the similarity pattern as a key factor to the performance gap, and introduce a metric, called Relative Fitting Difficulty (RFD), to measure the complexity of the similarity pattern. Then, based on the insights gained from the "What" question, we tackle the "How" question by increasing the pattern complexity of the training data. We achieve this by leveraging the In-Context Learning (ICL) capability of the Large Language Model (LLM) to generate data that simulates complex patterns. By utilizing the hierarchical patterns in the LLM-generated data, we effectively narrow the gap between supervised and unsupervised CSE. We release our codes and appendix at <a class="link-external link-https" href="https://github.com/BDBC-KG-NLP/NGCSE" rel="external noopener nofollow">this https URL</a>.

On training bi-directional neural network language model with noise contrastive estimation

Improved training of neural trans-dimensional random field language models with dynamic noise-contrastive estimation.

Data Noising as Smoothing in Neural Network Language Models

Learning neural trans-dimensional random field language models with noise-contrastive estimation

Learning Regularized Noise Contrastive Estimation for Robust Network Embedding.

Noise Contrastive Alignment of Language Models with Explicit Rewards

On the connection between Noise-Contrastive Estimation and Contrastive Divergence

Discriminative method for recurrent neural network language models

Contrastive and Consistency Learning for Neural Noisy-Channel Model in Spoken Language Understanding

Con-NAT: Contrastive Non-autoregressive Neural Machine Translation

Rnn-transducer with Language Bias for End-to-end Mandarin-English Code-switching Speech Recognition

Noise-robust Vision-language Pre-training with Positive-negative Learning

Asynchronous Bidirectional Decoding for Neural Machine Translation.

Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition

Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translation

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation

Curricular Contrastive Regularization for Speech Enhancement with Self-Supervised Representations.

Empirically Combining Unnormalized NNLM and Back-off N -Gram for Fast N -Best Rescoring in Speech Recognition

Recurrent Neural Network Based Language Model Adaptation for Accent Mandarin Speech.

Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model

CAPT: Contrastive Pre-Training for LearningDenoised Sequence Representations