Abstract:Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embedding space indicates closeness in the semantics between the sentences. Bilingual data offers a useful signal for learning such embeddings: properties shared by both sentences in a translation pair are likely semantic, while divergent properties are likely stylistic or language-specific. We propose a deep latent variable model that attempts to perform source separation on parallel sentences, isolating what they have in common in a latent semantic vector, and explaining what is left over with language-specific latent vectors. Our proposed approach differs from past work on semantic sentence encoding in two ways. First, by using a variational probabilistic framework, we introduce priors that encourage source separation, and can use our model's posterior to predict sentence embeddings for monolingual data at test time. Second, we use high-capacity transformers as both data generating distributions and inference networks -- contrasting with most past work on sentence embeddings. In experiments, our approach substantially outperforms the state-of-the-art on a standard suite of unsupervised semantic similarity evaluations. Further, we demonstrate that our approach yields the largest gains on more difficult subsets of these evaluations where simple word overlap is not a good indicator of similarity.

A Structured Variational Autoencoder for Contextual Morphological Inflection

Syntax-Directed Variational Autoencoder for Structured Data

Unsupervised Inflection Generation Using Neural Language Modeling

A Hierarchical Latent Structure for Variational Conversation Modeling

Linguistically inspired morphological inflection with a sequence to sequence model

Advanced Conditional Variational Autoencoders (A-CVAE): Towards interpreting open-domain conversation generation via disentangling latent feature representation

Disentangling Generative Factors in Natural Language with Discrete Variational Autoencoders

Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives

Adversarially Regularized Autoencoders

A Bilingual Generative Transformer for Semantic Sentence Embedding

A Variational Autoencoding Approach for Inducing Cross-lingual Word Embeddings

Improving Variational Autoencoders with Density Gap-based Regularization

A Latent Morphology Model for Open-Vocabulary Neural Machine Translation

Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space

Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational AutoEncoders

A Deep Generative Model for Code-Switched Text

Content Learning with Structure-Aware Writing: A Graph-Infused Dual Conditional Variational Autoencoder for Automatic Storytelling

Morphological Inflection with Phonological Features

VCDM: Leveraging Variational Bi-encoding and Deep Contextualized Word Representations for Improved Definition Modeling

On the Encoder-Decoder Incompatibility in Variational Text Modeling and Beyond

A Simple Joint Model for Improved Contextual Neural Lemmatization