Abstract:Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embedding space indicates closeness in the semantics between the sentences. Bilingual data offers a useful signal for learning such embeddings: properties shared by both sentences in a translation pair are likely semantic, while divergent properties are likely stylistic or language-specific. We propose a deep latent variable model that attempts to perform source separation on parallel sentences, isolating what they have in common in a latent semantic vector, and explaining what is left over with language-specific latent vectors. Our proposed approach differs from past work on semantic sentence encoding in two ways. First, by using a variational probabilistic framework, we introduce priors that encourage source separation, and can use our model's posterior to predict sentence embeddings for monolingual data at test time. Second, we use high-capacity transformers as both data generating distributions and inference networks -- contrasting with most past work on sentence embeddings. In experiments, our approach substantially outperforms the state-of-the-art on a standard suite of unsupervised semantic similarity evaluations. Further, we demonstrate that our approach yields the largest gains on more difficult subsets of these evaluations where simple word overlap is not a good indicator of similarity.

A Novel Bilingual Word Embedding Method for Lexical Translation Using Bilingual Sense Clique

Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context

Learning Sense-specific Word Embeddings By Exploiting Bilingual Resources.

Bilingual word embedding fusion for robust unsupervised bilingual lexicon induction

Learning Chinese-Japanese Bilingual Word Embedding by Using Common Characters.

Discovering Bilingual Lexicons in Polyglot Word Embeddings

A Bilingual Generative Transformer for Semantic Sentence Embedding

Bilingual Sentiment Embeddings: Joint Projection of Sentiment Across Languages

Improve Statistical Machine Translation with Context-Sensitive Bilingual Semantic Embedding Model.

Shared-Private Bilingual Word Embeddings for Neural Machine Translation

Multi-phase Word Sense Embedding Learning Using a Corpus and a Lexical Ontology.

Dual-perspective fusion for word translation enhancement

A Comparison of Word Embeddings for English and Cross-Lingual Chinese Word Sense Disambiguation

Learning Bilingual Embedding Model for Cross-Language Sentiment Classification

Unsupervised Bilingual Lexicon Induction Via Latent Variable Models.

Bilingual Lexicon Induction from Non-Parallel Data with Minimal Supervision.

Jointly Learning Bilingual Word Embeddings and Alignments

Multi-phase Word Sense Embedding Retrofitting with Lexical Ontology

Beyond Shared Vocabulary: Increasing Representational Word Similarities across Languages for Multilingual Machine Translation

Building Earth Mover'S Distance on Bilingual Word Embeddings for Machine Translation

Bilingual Embeddings with Random Walks over Multilingual Wordnets