Abstract:Recent state-of-the-art natural language understanding models, such as BERT and XLNet, score a pair of sentences (A and B) using multiple cross-attention operations - a process in which each word in sentence A attends to all words in sentence B and vice versa. As a result, computing the similarity between a query sentence and a set of candidate sentences, requires the propagation of all query-candidate sentence-pairs throughout a stack of cross-attention layers. This exhaustive process becomes computationally prohibitive when the number of candidate sentences is large. In contrast, sentence embedding techniques learn a sentence-to-vector mapping and compute the similarity between the sentence vectors via simple elementary operations. In this paper, we introduce Distilled Sentence Embedding (DSE) - a model that is based on knowledge distillation from cross-attentive models, focusing on sentence-pair tasks. The outline of DSE is as follows: Given a cross-attentive teacher model (e.g. a fine-tuned BERT), we train a sentence embedding based student model to reconstruct the sentence-pair scores obtained by the teacher model. We empirically demonstrate the effectiveness of DSE on five GLUE sentence-pair tasks. DSE significantly outperforms several ELMO variants and other sentence embedding methods, while accelerating computation of the query-candidate sentence-pairs similarities by several orders of magnitude, with an average relative degradation of 4.6% compared to BERT. Furthermore, we show that DSE produces sentence embeddings that reach state-of-the-art performance on universal sentence representation benchmarks. Our code is made publicly available at <a class="link-external link-https" href="https://github.com/microsoft/Distilled-Sentence-Embedding" rel="external noopener nofollow">this https URL</a>.

Sentence Pair Similarity Modeling Based on Weighted Interaction of Multi-semantic Embedding Matrix

Inter-Weighted Alignment Network for Sentence Pair Modeling.

A Novel Sentence Similarity Model with Word Embedding Based on Convolutional Neural Network.

A Joint Model for Sentence Semantic Similarity Learning

Semantic Similarity Computing Model Based on Multi Model Fine-Grained Nonlinear Fusion

SentPWNet: A Unified Sentence Pair Weighting Network for Task-specific Sentence Embedding

A Model for Chinese Sentence Similarity Computing

Sentence-Embedding and Similarity via Hybrid Bidirectional-LSTM and CNN Utilizing Weighted-Pooling Attention

Chinese Sentence Similarity Computational Model Based on Multi-Features Combination

An Efficient Framework for Sentence Similarity Modeling.

Model for Sentence Similarity Computing Based on Multi-Features Combination

Short text matching model with multiway semantic interaction based on multi-granularity semantic embedding

MNet-Sim: A Multi-layered Semantic Similarity Network to Evaluate Sentence Similarity

A Comparative Study of Sentence Embedding Models for Assessing Semantic Variation

Scalable Attentive Sentence-Pair Modeling via Distilled Sentence Embedding

Sentence Similarity Computation by Integrating Shallow and Deep Information

Sentence Similarity Based on Contexts

Sentence Similarity Computation in Question Answering Robot

Modelling Interaction of Sentence Pair with Coupled-LSTMs.

Chinese Sentence Similarity Based on Multi-feature Combination

Chinese Sentence Similarity Measure Based on Word Sequence Length and Word Weight