Abstract:Multi-lingual language models (LM), such as mBERT, XLM-R, mT5, mBART, have been remarkably successful in enabling natural language tasks in low-resource languages through cross-lingual transfer from high-resource ones. In this work, we try to better understand how such models, specifically mT5, transfer *any* linguistic and semantic knowledge across languages, even though no explicit cross-lingual signals are provided during pre-training. Rather, only unannotated texts from each language are presented to the model separately and independently of one another, and the model appears to implicitly learn cross-lingual connections. This raises several questions that motivate our study, such as: Are the cross-lingual connections between every language pair equally strong? What properties of source and target language impact the strength of cross-lingual transfer? Can we quantify the impact of those properties on the cross-lingual transfer? In our investigation, we analyze a pre-trained mT5 to discover the attributes of cross-lingual connections learned by the model. Through a statistical interpretation framework over 90 language pairs across three tasks, we show that transfer performance can be modeled by a few linguistic and data-derived features. These observations enable us to interpret cross-lingual understanding of the mT5 model. Through these observations, one can favorably choose the best source language for a task, and can anticipate its training data demands. A key finding of this work is that similarity of syntax, morphology and phonology are good predictors of cross-lingual transfer, significantly more than just the lexical similarity of languages. For a given language, we are able to predict zero-shot performance, that increases on a logarithmic scale with the number of few-shot target language data points.

Improving Similar Language Translation With Transfer Learning

Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation

Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task

Low Resource Arabic Dialects Transformer Neural Machine Translation Improvement through Incremental Transfer of Shared Linguistic Features

Improved Cross-Lingual Transfer Learning For Automatic Speech Translation

Bilingual Dictionary-based Language Model Pretraining for Neural Machine Translation

Machine Translation of Low-Resource Indo-European Languages

Beyond Shared Vocabulary: Increasing Representational Word Similarities across Languages for Multilingual Machine Translation

Languages You Know Influence Those You Learn: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfer

Improving Multilingual Semantic Textual Similarity with Shared Sentence Encoder for Low-resource Languages

Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis

Learning Language Specific Sub-network for Multilingual Machine Translation

Improving Polish to English Neural Machine Translation with Transfer Learning: Effects of Data Volume and Language Similarity

Multilingual Transfer and Domain Adaptation for Low-Resource Languages of Spain

Improving Neural Machine Translation Models with Monolingual Data

From English To Foreign Languages: Transferring Pre-trained Language Models

Neural Machine Translation into Language Varieties

Machine Translation Advancements of Low-Resource Indian Languages by Transfer Learning

Scaling neural machine translation to 200 languages

No Language Left Behind: Scaling Human-Centered Machine Translation

Dual Transfer Learning for Neural Machine Translation with Marginal Distribution Regularization