Abstract:Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embedding space indicates closeness in the semantics between the sentences. Bilingual data offers a useful signal for learning such embeddings: properties shared by both sentences in a translation pair are likely semantic, while divergent properties are likely stylistic or language-specific. We propose a deep latent variable model that attempts to perform source separation on parallel sentences, isolating what they have in common in a latent semantic vector, and explaining what is left over with language-specific latent vectors. Our proposed approach differs from past work on semantic sentence encoding in two ways. First, by using a variational probabilistic framework, we introduce priors that encourage source separation, and can use our model's posterior to predict sentence embeddings for monolingual data at test time. Second, we use high-capacity transformers as both data generating distributions and inference networks -- contrasting with most past work on sentence embeddings. In experiments, our approach substantially outperforms the state-of-the-art on a standard suite of unsupervised semantic similarity evaluations. Further, we demonstrate that our approach yields the largest gains on more difficult subsets of these evaluations where simple word overlap is not a good indicator of similarity.

Semantic Sentence Embeddings for Paraphrasing and Text Summarization

Automatic Document Summarization Via Deep Neural Networks

Paraphrase Thought: Sentence Embedding Module Imitating Human Language Recognition

A Semantically Consistent and Syntactically Variational Encoder-Decoder Framework for Paraphrase Generation.

Semantic Vector Machines

Text Summarization Based on Sentence Selection with Semantic Representation

Latent Semantic Analysis Approach for Document Summarization Based on Word Embeddings

Generating Sentences from Disentangled Syntactic and Semantic Spaces

Semantic Hilbert Space for Text Representation Learning

A Bilingual Generative Transformer for Semantic Sentence Embedding

Global Encoding for Abstractive Summarization

Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations

Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representations

Neural Summarization by Extracting Sentences and Words

Query and Output: Generating Words by Querying Distributed Word Representations for Paraphrase Generation

Sentence Vector Model Based on Implicit Word Vector Expression

Selective Encoding for Abstractive Sentence Summarization

A Deep Generative Framework for Paraphrase Generation

Bipartite Graph Pre-training for Unsupervised Extractive Summarization with Graph Convolutional Auto-Encoders

Learning semantic sentence representations from visually grounded language without lexical knowledge

Weakly-Supervised Video Summarization Using Variational Encoder-Decoder And Web Prior