Abstract:Learning semantic sentence embeddings is beneficial to a variety of natural language processing tasks. Recently, methods using the contrastive learning framework to fine-tune pre-trained language models have been proposed and have achieved significant performance on sentence embeddings. However, sentence embeddings are easy to "overfit" to the contrastive learning goal. With the training of contrastive learning, the gap between contrastive learning and test tasks leads to unstable even declining performance on test tasks. For this reason, existing methods rely on the labeled development set to frequently evaluate the performance on test tasks and get the best checkpoints. In such a way, models are limited when the labeled data is unavailable or extremely scarce. To address this problem, we propose Pseudo-Siamese network Mutual Learning (PSML) for self-supervised sentence embeddings to reduce the gap between contrastive learning and test tasks. Consisting of the main encoder and the auxiliary encoder, PSML utilizes mutual learning as the basic framework. Between the two encoders, two mutual learning losses are constructed to share learning signals. The proposed model framework and losses of PSML help the model be optimized more stably and generalize better to test tasks, such as semantic textual similarity. Extensive experiments on seven public semantic textual similarity datasets show that PSML performs better than previous unsupervised contrastive methods for sentence embeddings. Besides, PSML also gives a stable performance curve on test tasks with training and is able to get the comparative performance without frequent evaluation on the labeled development set.

Learning Multilingual Sentence Embeddings From Monolingual Corpus

Unsupervised Cross-Lingual Sentence Representation Learning via Linguistic Isomorphism

Exploring Multilingual Syntactic Sentence Representations

Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax

Revisiting Language Encoding in Learning Multilingual Representations

Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embedding

Bilingual word embedding fusion for robust unsupervised bilingual lexicon induction

Jointly Learning Bilingual Word Embeddings and Alignments

Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context

Obtaining Parallel Sentences in Low-Resource Language Pairs with Minimal Supervision

Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora

A Bilingual Generative Transformer for Semantic Sentence Embedding

Learning Multilingual Sentence Representations with Cross-lingual Consistency Regularization

Joint Bilingual Sentiment Classification with Unlabeled Parallel Corpora.

Linear Cross-Lingual Mapping of Sentence Embeddings

Learning Bilingual Sentiment-Specific Word Embeddings without Cross-Lingual Supervision

A Mutually Reinforced Framework for Pretrained Sentence Embeddings

Learning Sense-specific Word Embeddings By Exploiting Bilingual Resources.

Learning Cross-lingual Word Embeddings Via Matrix Co-factorization.

Stable Contrastive Learning for Self-Supervised Sentence Embeddings With Pseudo-Siamese Mutual Learning

Adversarial Training for Unsupervised Bilingual Lexicon Induction