Abstract:Pretrained language models have achieved great success in a wide range of natural language processing (NLP) problems, because they learn language representations from large-scale text corpora and can adapt to downstream tasks by finetuning them on annotated task data. However, such success relies on both large-scale text and annotated data, so the lack of training data is a major practical problem for many languages, especially low-resource languages. In this paper, we explore whether a pretrained English language model can benefit non-English NLP systems in low-resource scenarios, i.e., with limited text corpora or annotated data. To achieve this, we first propose cross-lingual knowledge transfer methods and then validate our methods in low-resource scenarios. Specifically, our cross-lingual knowledge transfer methods are applied in the training stages of language model pretraining or downstream finetuning. At the two stages, the methods are designed for the transfer of upstream general knowledge or downstream task-specific knowledge, respectively. In the experiments, we perform pretraining and finetuning with limited non-English data to simulate the low-resource scenarios. We evaluate our methods on ten downstream tasks over a wide range of languages, and present systematic comparisons among various knowledge transfer methods. Experimental results show that our methods successfully leverage a pretrained English language model to improve task performance in other languages. Besides, we demonstrate the multilinguality of the English language model in various application scenarios. Our findings imply the possibility to improve low-resource-language NLP systems with large-scale English language models.

Cross-Lingual Language Model Meta-Pretraining

Cross-Lingual Supervision improves Large Language Models Pre-training

Multimodal Pretraining from Monolingual to Multilingual

Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training

INFOXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

Cross-Lingual Natural Language Generation via Pre-Training

Can Monolingual Pretrained Models Help Cross-Lingual Classification?

Alternating Language Modeling for Cross-Lingual Pre-Training.

Analyzing the mono-and cross-lingual pretraining dynamics of multilingual language models

Cross-lingual Adaption Model-Agnostic Meta-Learning for Natural Language Understanding

Unifying Cross-Lingual and Cross-Modal Modeling Towards Weakly Supervised Multilingual Vision-Language Pre-training

VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation

Emerging cross-lingual structure in pretrained language models

Meta Distant Transfer Learning for Pre-trained Language Models.

Can Pretrained English Language Models Benefit Non-English NLP Systems in Low-Resource Scenarios?

mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models?

MULTI-LEVEL CONTRASTIVE LEARNING FOR CROSS-LINGUAL ALIGNMENT

Meta-adapter: efficient cross-lingual adaptation with meta-learning

Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly

Multilingual Pre-training with Universal Dependency Learning.

Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation