Abstract:Recent state-of-the-art natural language understanding models, such as BERT and XLNet, score a pair of sentences (A and B) using multiple cross-attention operations - a process in which each word in sentence A attends to all words in sentence B and vice versa. As a result, computing the similarity between a query sentence and a set of candidate sentences, requires the propagation of all query-candidate sentence-pairs throughout a stack of cross-attention layers. This exhaustive process becomes computationally prohibitive when the number of candidate sentences is large. In contrast, sentence embedding techniques learn a sentence-to-vector mapping and compute the similarity between the sentence vectors via simple elementary operations. In this paper, we introduce Distilled Sentence Embedding (DSE) - a model that is based on knowledge distillation from cross-attentive models, focusing on sentence-pair tasks. The outline of DSE is as follows: Given a cross-attentive teacher model (e.g. a fine-tuned BERT), we train a sentence embedding based student model to reconstruct the sentence-pair scores obtained by the teacher model. We empirically demonstrate the effectiveness of DSE on five GLUE sentence-pair tasks. DSE significantly outperforms several ELMO variants and other sentence embedding methods, while accelerating computation of the query-candidate sentence-pairs similarities by several orders of magnitude, with an average relative degradation of 4.6% compared to BERT. Furthermore, we show that DSE produces sentence embeddings that reach state-of-the-art performance on universal sentence representation benchmarks. Our code is made publicly available at <a class="link-external link-https" href="https://github.com/microsoft/Distilled-Sentence-Embedding" rel="external noopener nofollow">this https URL</a>.

Sebgm: Sentence Embedding Based on Generation Model with Multi-Task Learning

OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding

Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework

Going Beyond Multi-Task Dense Prediction with Synergy Embedding Models

Do Multi-Sense Embeddings Improve Natural Language Understanding?

Unsupervised Sentence Embedding Model Based on Contrastive Learning

Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learning

CMLM-CSE: Based on Conditional MLM Contrastive Learning for Sentence Embeddings

Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings

Simple Flow-Based Contrastive Learning for BERT Sentence Representations

Towards General Text Embeddings with Multi-stage Contrastive Learning

Simple Techniques for Enhancing Sentence Embeddings in Generative Language Models

InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings

Leveraging Balanced Semantic Embedding for Generative Zero-Shot Learning

A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddings

CLSEP: Contrastive learning of sentence embedding with prompt

Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning

Stable Contrastive Learning for Self-Supervised Sentence Embeddings With Pseudo-Siamese Mutual Learning

Contrastive Learning Models for Sentence Representations

UNSEE: Unsupervised Non-contrastive Sentence Embeddings

Scalable Attentive Sentence-Pair Modeling via Distilled Sentence Embedding