Abstract:Recent state-of-the-art natural language understanding models, such as BERT and XLNet, score a pair of sentences (A and B) using multiple cross-attention operations - a process in which each word in sentence A attends to all words in sentence B and vice versa. As a result, computing the similarity between a query sentence and a set of candidate sentences, requires the propagation of all query-candidate sentence-pairs throughout a stack of cross-attention layers. This exhaustive process becomes computationally prohibitive when the number of candidate sentences is large. In contrast, sentence embedding techniques learn a sentence-to-vector mapping and compute the similarity between the sentence vectors via simple elementary operations. In this paper, we introduce Distilled Sentence Embedding (DSE) - a model that is based on knowledge distillation from cross-attentive models, focusing on sentence-pair tasks. The outline of DSE is as follows: Given a cross-attentive teacher model (e.g. a fine-tuned BERT), we train a sentence embedding based student model to reconstruct the sentence-pair scores obtained by the teacher model. We empirically demonstrate the effectiveness of DSE on five GLUE sentence-pair tasks. DSE significantly outperforms several ELMO variants and other sentence embedding methods, while accelerating computation of the query-candidate sentence-pairs similarities by several orders of magnitude, with an average relative degradation of 4.6% compared to BERT. Furthermore, we show that DSE produces sentence embeddings that reach state-of-the-art performance on universal sentence representation benchmarks. Our code is made publicly available at <a class="link-external link-https" href="https://github.com/microsoft/Distilled-Sentence-Embedding" rel="external noopener nofollow">this https URL</a>.

Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features

GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings

Learning Multilingual Sentence Embeddings From Monolingual Corpus

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations

UNSEE: Unsupervised Non-contrastive Sentence Embeddings

Ranking-Enhanced Unsupervised Sentence Representation Learning

Compositional Network Embedding.

Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework

Training and Evaluating Improved Dependency-Based Word Embeddings.

Unsupervised POS Induction with Word Embeddings

An Unsupervised Sentence Embedding Method by Mutual Information Maximization

Evaluation of Unsupervised Compositional Representations

Improving Compositional Generalization Using Iterated Learning and Simplicial Embeddings

An Unsupervised Sentence Embedding Method Bymutual Information Maximization.

Unsupervised Sentence Embedding Model Based on Contrastive Learning

Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization

Multimodal Deep Embedding via Hierarchical Grounded Compositional Semantics.

Scalable Attentive Sentence-Pair Modeling via Distilled Sentence Embedding

Unsupervised Morphological Expansion of Small Datasets for Improving Word Embeddings

Semantic Sentence Embeddings for Paraphrasing and Text Summarization