Abstract:Unsupervised word representations have demonstrated improvements in predictive generalization on various NLP tasks. Most of the existing models are in fact good at capturing the relatedness among words rather than their "genuine" similarity because the context representations are often represented by a sum (or an average) of the neighbor's embeddings, which simplifies the computation but ignores an important fact that the meaning of a word is determined by its context, reflecting not only the surrounding words but also the rules used to combine them (i.e. compositionality). On the other hand, much effort has been devoted to learning a single prototype representation per word, which is problematic because many words are polysemous, and a single-prototype model is incapable of capturing phenomena of homonymy and polysemy. We present a neural network architecture to jointly learn word embeddings and context representations from large data sets. The explicitly produced context representations are further used to learn context-specific and multi prototype word embeddings. Our embeddings were evaluated on several NLP tasks, and the experimental results demonstrated the proposed model outperformed other competitors and is applicable to intrinsically "character-based" languages.

Context-Aware Phrase Representation For Statistical Machine Translation

Topically-informed Bilingually-Constrained Recursive Autoencoders for Statistical Machine Translation

Learning Bilingual Distributed Phrase Represenations for Statistical Machine Translation

Syntax-Based Context Representation for Statistical Machine Translation.

Translating Phrases in Neural Machine Translation

A Context-Aware Topic Model for Statistical Machine Translation.

Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation

Learning High-Quality and General-Purpose Phrase Representations

Improve Statistical Machine Translation with Context-Sensitive Bilingual Semantic Embedding Model.

Learning Semantic Representations for Nonterminals in Hierarchical Phrase-Based Translation.

An Improved Hierarchical Phrase Based Machine Translation Model

A Neural Approach to Source Dependence Based Context Model for Statistical Machine Translation

A Topic Similarity Model for Hierarchical Phrase-based Translation.

Learning Context-Specific Word/Character Embeddings.

Learning word representation by jointly using neighbor and syntactic contexts

Learning Contextualized Sentence Representations for Document-Level Neural Machine Translation

Syntax Encapsulated Phrase Model for Statistical Machine Translation

Translation Prediction with Source Dependency-Based Context Representation.

A Neural Reordering Model for Phrase-based Translation.

Phrase-Based & Neural Unsupervised Machine Translation

A Content-Based Neural Reordering Model for Statistical Machine Translation