Abstract:Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embedding space indicates closeness in the semantics between the sentences. Bilingual data offers a useful signal for learning such embeddings: properties shared by both sentences in a translation pair are likely semantic, while divergent properties are likely stylistic or language-specific. We propose a deep latent variable model that attempts to perform source separation on parallel sentences, isolating what they have in common in a latent semantic vector, and explaining what is left over with language-specific latent vectors. Our proposed approach differs from past work on semantic sentence encoding in two ways. First, by using a variational probabilistic framework, we introduce priors that encourage source separation, and can use our model's posterior to predict sentence embeddings for monolingual data at test time. Second, we use high-capacity transformers as both data generating distributions and inference networks -- contrasting with most past work on sentence embeddings. In experiments, our approach substantially outperforms the state-of-the-art on a standard suite of unsupervised semantic similarity evaluations. Further, we demonstrate that our approach yields the largest gains on more difficult subsets of these evaluations where simple word overlap is not a good indicator of similarity.

A Variational Autoencoding Approach for Inducing Cross-lingual Word Embeddings

Bilingual Autoencoders with Global Descriptors for Modeling Parallel Sentences.

A Neural Generative Autoencoder for Bilingual Word Embeddings.

Unsupervised Bilingual Lexicon Induction Via Latent Variable Models.

A Bilingual Generative Transformer for Semantic Sentence Embedding

Exploring Implicit Semantic Constraints for Bilingual Word Embeddings

On the Encoder-Decoder Incompatibility in Variational Text Modeling and Beyond

Improving Vector Space Word Representations Via Kernel Canonical Correlation Analysis

A Bilingual Adversarial Autoencoder for Unsupervised Bilingual Lexicon Induction

Dual-View Variational Autoencoders for Semi-Supervised Text Matching

Learning Tibetan-Chinese Cross-Lingual Word Embeddings

Unsupervised Cross-Lingual Sentence Representation Learning via Linguistic Isomorphism

Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context

Learning Multilingual Topics with Neural Variational Inference

Jointly Learning Bilingual Word Embeddings and Alignments

Narrowing the Variance of Variational Cross-Encoder for Cross-Modal Hashing

Bilingual Lexicon Induction from Non-Parallel Data with Minimal Supervision.

Disentangling Generative Factors in Natural Language with Discrete Variational Autoencoders

Adversarial Training for Unsupervised Bilingual Lexicon Induction

VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation

Cross-lingual Models of Word Embeddings: An Empirical Comparison