Abstract:Learning semantic sentence embeddings is beneficial to a variety of natural language processing tasks. Recently, methods using the contrastive learning framework to fine-tune pre-trained language models have been proposed and have achieved significant performance on sentence embeddings. However, sentence embeddings are easy to "overfit" to the contrastive learning goal. With the training of contrastive learning, the gap between contrastive learning and test tasks leads to unstable even declining performance on test tasks. For this reason, existing methods rely on the labeled development set to frequently evaluate the performance on test tasks and get the best checkpoints. In such a way, models are limited when the labeled data is unavailable or extremely scarce. To address this problem, we propose Pseudo-Siamese network Mutual Learning (PSML) for self-supervised sentence embeddings to reduce the gap between contrastive learning and test tasks. Consisting of the main encoder and the auxiliary encoder, PSML utilizes mutual learning as the basic framework. Between the two encoders, two mutual learning losses are constructed to share learning signals. The proposed model framework and losses of PSML help the model be optimized more stably and generalize better to test tasks, such as semantic textual similarity. Extensive experiments on seven public semantic textual similarity datasets show that PSML performs better than previous unsupervised contrastive methods for sentence embeddings. Besides, PSML also gives a stable performance curve on test tasks with training and is able to get the comparative performance without frequent evaluation on the labeled development set.

Unsupervised Sentence Embedding Model Based on Contrastive Learning

OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding

SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives

InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings

DebCSE: Rethinking Unsupervised Contrastive Sentence Embedding Learning in the Debiasing Perspective

DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings

UNSEE: Unsupervised Non-contrastive Sentence Embeddings

Sparse Contrastive Learning of Sentence Embeddings

Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework

CMLM-CSE: Based on Conditional MLM Contrastive Learning for Sentence Embeddings

CLSESSP: Contrastive learning of sentence embedding with strong semantic prototypes

Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings

Simple Flow-Based Contrastive Learning for BERT Sentence Representations

A Mutually Reinforced Framework for Pretrained Sentence Embeddings

CCDC: A Chinese-Centric Cross Domain Contrastive Learning Framework

Stable Contrastive Learning for Self-Supervised Sentence Embeddings With Pseudo-Siamese Mutual Learning

A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddings

Text Semantic Matching with an Enhanced Sample Building Method Based on Contrastive Learning

RankCSE: Unsupervised Sentence Representations Learning Via Learning to Rank

SCD: Self-Contrastive Decorrelation for Sentence Embeddings

Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embedding