Abstract:Multi-lingual language models (LM), such as mBERT, XLM-R, mT5, mBART, have been remarkably successful in enabling natural language tasks in low-resource languages through cross-lingual transfer from high-resource ones. In this work, we try to better understand how such models, specifically mT5, transfer *any* linguistic and semantic knowledge across languages, even though no explicit cross-lingual signals are provided during pre-training. Rather, only unannotated texts from each language are presented to the model separately and independently of one another, and the model appears to implicitly learn cross-lingual connections. This raises several questions that motivate our study, such as: Are the cross-lingual connections between every language pair equally strong? What properties of source and target language impact the strength of cross-lingual transfer? Can we quantify the impact of those properties on the cross-lingual transfer? In our investigation, we analyze a pre-trained mT5 to discover the attributes of cross-lingual connections learned by the model. Through a statistical interpretation framework over 90 language pairs across three tasks, we show that transfer performance can be modeled by a few linguistic and data-derived features. These observations enable us to interpret cross-lingual understanding of the mT5 model. Through these observations, one can favorably choose the best source language for a task, and can anticipate its training data demands. A key finding of this work is that similarity of syntax, morphology and phonology are good predictors of cross-lingual transfer, significantly more than just the lexical similarity of languages. For a given language, we are able to predict zero-shot performance, that increases on a logarithmic scale with the number of few-shot target language data points.

Marvelous Agglutinative Language Effect on Cross Lingual Transfer Learning

Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean

Code-Switching Curriculum Learning for Multilingual Transfer in LLMs

Analysis of Multi-Source Language Training in Cross-Lingual Transfer

Cross-lingual Transfer of Monolingual Models

Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabularies

X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment

Accelerating Multilingual Language Model for Excessively Tokenized Languages

Mitigating the Linguistic Gap with Phonemic Representations for Robust Cross-lingual Transfer

An Efficient Approach for Studying Cross-Lingual Transfer in Multilingual Language Models

Languages You Know Influence Those You Learn: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfer

Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models

Optimizing Two-Pass Cross-Lingual Transfer Learning: Phoneme Recognition and Phoneme to Grapheme Translation

DaG LLM ver 1.0: Pioneering Instruction-Tuned Language Modeling for Korean NLP

Korean-English Machine Translation with Multiple Tokenization Strategy

Choosing Transfer Languages for Cross-Lingual Learning

Ancient Korean Archive Translation: Comparison Analysis on Statistical phrase alignment, LLM in-context learning, and inter-methodological approach

mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models?

Analysing The Impact Of Linguistic Features On Cross-Lingual Transfer

How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning

Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment