Abstract:Despite the recent success of deep neural networks in natural language processing (NLP), their interpretability remains a challenge. We analyze the representations learned by neural machine translation models at various levels of granularity and evaluate their quality through relevant extrinsic properties. In particular, we seek answers to the following questions: (i) How accurately is word-structure captured within the learned representations, an important aspect in translating morphologically-rich languages? (ii) Do the representations capture long-range dependencies, and effectively handle syntactically divergent languages? (iii) Do the representations capture lexical semantics? We conduct a thorough investigation along several parameters: (i) Which layers in the architecture capture each of these linguistic phenomena; (ii) How does the choice of translation unit (word, character, or subword unit) impact the linguistic properties captured by the underlying representations? (iii) Do the encoder and decoder learn differently and independently? (iv) Do the representations learned by multilingual NMT models capture the same amount of linguistic information as their bilingual counterparts? Our data-driven, quantitative evaluation illuminates important aspects in NMT models and their ability to capture various linguistic phenomena. We show that deep NMT models learn a non-trivial amount of linguistic information. Notable findings include: i) Word morphology and part-of-speech information are captured at the lower layers of the model; (ii) In contrast, lexical semantics or non-local syntactic and semantic dependencies are better represented at the higher layers; (iii) Representations learned using characters are more informed about wordmorphology compared to those learned using subword units; and (iv) Representations learned by multilingual models are richer compared to bilingual models.

Are Mutually Intelligible Languages Easier to Translate?

A Computational Model for the Assessment of Mutual Intelligibility Among Closely Related Languages

Mutual Information and Diverse Decoding Improve Neural Machine Translation.

Are All Languages Equally Hard to Language-Model?

It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information

A Theory of Unsupervised Translation Motivated by Understanding Animal Communication

Could We Have Had Better Multilingual LLMs If English Was Not the Central Language?

What Kind of Language Is Hard to Language-Model?

Causes and Cures for Interference in Multilingual Translation

Scaling Laws for Multilingual Neural Machine Translation

Improving Language Model Integration for Neural Machine Translation

On the Linguistic Representational Power of Neural Machine Translation Models

Embedding Word Similarity with Neural Machine Translation

On the Integration of LinguisticFeatures into Statistical and Neural Machine Translation

Emerging Language Spaces Learned From Massively Multilingual Corpora

Exploring the Correlation between Human and Machine Evaluation of Simultaneous Speech Translation

On Learning Language-Invariant Representations for Universal Machine Translation

What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages

Machine Translation of Low-Resource Indo-European Languages

The Less the Merrier? Investigating Language Representation in Multilingual Models

Languages You Know Influence Those You Learn: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfer