Abstract:Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embedding space indicates closeness in the semantics between the sentences. Bilingual data offers a useful signal for learning such embeddings: properties shared by both sentences in a translation pair are likely semantic, while divergent properties are likely stylistic or language-specific. We propose a deep latent variable model that attempts to perform source separation on parallel sentences, isolating what they have in common in a latent semantic vector, and explaining what is left over with language-specific latent vectors. Our proposed approach differs from past work on semantic sentence encoding in two ways. First, by using a variational probabilistic framework, we introduce priors that encourage source separation, and can use our model's posterior to predict sentence embeddings for monolingual data at test time. Second, we use high-capacity transformers as both data generating distributions and inference networks -- contrasting with most past work on sentence embeddings. In experiments, our approach substantially outperforms the state-of-the-art on a standard suite of unsupervised semantic similarity evaluations. Further, we demonstrate that our approach yields the largest gains on more difficult subsets of these evaluations where simple word overlap is not a good indicator of similarity.

Bilingual Learning of Multi-sense Embeddings with Discrete Autoencoders

Do Multi-Sense Embeddings Improve Natural Language Understanding?

Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context

Going Beyond Multi-Task Dense Prediction with Synergy Embedding Models

Learning Sense-specific Word Embeddings By Exploiting Bilingual Resources.

A Bilingual Generative Transformer for Semantic Sentence Embedding

Learning Multilingual Sentence Embeddings From Monolingual Corpus

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Unsupervised Multimodal Language Representations using Convolutional Autoencoders

Together We Make Sense -- Learning Meta-Sense Embeddings from Pretrained Static Sense Embeddings

Learning Word Sense Embeddings from Word Sense Definitions

A Variational Autoencoding Approach for Inducing Cross-lingual Word Embeddings

Natural Language Multitasking: Analyzing and Improving Syntactic Saliency of Hidden Representations

LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond

A Novel Bilingual Word Embedding Method for Lexical Translation Using Bilingual Sense Clique

Bilingual Sentiment Embeddings: Joint Projection of Sentiment Across Languages

Semi-Supervised Learning for Bilingual Lexicon Induction

Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax

Learning to Predict: A Fast Re-constructive Method to Generate Multimodal Embeddings

Distributional Models and Deep Learning Embeddings: Combining the Best of Both Worlds

Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation