Abstract:Recent multilingual pretrained language models (mPLMs) often avoid using language embeddings -- learnable vectors assigned to different languages. These embeddings are discarded for two main reasons: (1) mPLMs are expected to have a single, unified parameter set across all languages, and (2) they need to function seamlessly as universal text encoders without requiring language IDs as input. However, this removal increases the burden on token embeddings to encode all language-specific information, which may hinder the model's ability to produce more language-neutral representations. To address this challenge, we propose Language-Script Aware Multilingual Pretraining (LangSAMP), a method that incorporates both language and script embeddings to enhance representation learning while maintaining a simple architecture. Specifically, we integrate these embeddings into the output of the transformer blocks before passing the final representations to the language modeling head for prediction. We apply LangSAMP to the continual pretraining of XLM-R on a highly multilingual corpus covering more than 500 languages. The resulting model consistently outperforms the baseline. Extensive analysis further shows that language/script embeddings encode language/script-specific information, which improves the selection of source languages for crosslingual transfer. We make our code and models publicly available at \url{<a class="link-external link-https" href="https://github.com/cisnlp/LangSAMP" rel="external noopener nofollow">this https URL</a>}.

Discovering Representation Sprachbund for Multilingual Pre-Training

Pre-training Universal Language Representation

Multimodal Pretraining from Monolingual to Multilingual

Multilingual Translation with Extensible Multilingual Pretraining and Finetuning

Discovering Language-neutral Sub-networks in Multilingual Language Models

Multilingual Pre-training with Universal Dependency Learning.

Learning Multilingual Representation for Natural Language Understanding with Enhanced Cross-Lingual Supervision

Unsupervised cross-lingual representation learning at scale

Emerging cross-lingual structure in pretrained language models

Generalizing Multimodal Pre-training into Multilingual via Language Acquisition

Pre-trained Language Model Representations for Language Generation

Unifying Cross-Lingual and Cross-Modal Modeling Towards Weakly Supervised Multilingual Vision-Language Pre-training

Soft Language Clustering for Multilingual Model Pre-training

Improving Pre-Trained Multilingual Model with Vocabulary Expansion

INFOXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

Multi-Lingual Acquisition on Multimodal Pre-training for Cross-modal Retrieval

Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment

A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectives

LangSAMP: Language-Script Aware Multilingual Pretraining

Cross-Lingual Self-training to Learn Multilingual Representation for Low-Resource Speech Recognition

Leveraging Multilingual Self-Supervised Pretrained Models for Sequence-to-Sequence End-to-End Spoken Language Understanding