Abstract:We propose a novel scaling law for general-purpose decoder-only language models (LMs) trained on multilingual data, addressing the problem of balancing languages during multilingual pretraining. A primary challenge in studying multilingual scaling is the difficulty of analyzing individual language performance due to cross-lingual transfer. To address this, we shift the focus from individual languages to language families. We introduce and validate a hypothesis that the test cross-entropy loss for each language family is determined solely by its own sampling ratio, independent of other languages in the mixture. This insight simplifies the complexity of multilingual scaling and make the analysis scalable to an arbitrary number of languages. Building on this hypothesis, we derive a power-law relationship that links performance with dataset size, model size and sampling ratios. This relationship enables us to predict performance across various combinations of the above three quantities, and derive the optimal sampling ratios at different model scales. To demonstrate the effectiveness and accuracy of our proposed scaling law, we perform a large-scale empirical study, training more than 100 models on 23 languages spanning 5 language families. Our experiments show that the optimal sampling ratios derived from small models (85M parameters) generalize effectively to models that are several orders of magnitude larger (1.2B parameters), offering a resource-efficient approach for multilingual LM training at scale.

Adapting BigScience Multilingual Model to Unseen Languages

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains.

Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation

An Empirical Study of Factors Affecting Language-Independent Models

Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability

Bilingual Adaptation of Monolingual Foundation Models

Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages

Training Bilingual LMs with Data Constraints in the Targeted Language

Methodology of Adapting Large English Language Models for Specific Cultural Contexts

Larger-Scale Transformers for Multilingual Masked Language Modeling

Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions?

Targeted Multilingual Adaptation for Low-resource Language Families

Scaling Laws for Multilingual Neural Machine Translation

From English To Foreign Languages: Transferring Pre-trained Language Models

Scaling Laws for Multilingual Language Models

LangBridge: Multilingual Reasoning Without Multilingual Supervision

Are Pretrained Multilingual Models Equally Fair Across Languages?

Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages

On the Limitations of Sociodemographic Adaptation with Transformers

SambaLingo: Teaching Large Language Models New Languages

A Recipe of Parallel Corpora Exploitation for Multilingual Large Language Models