Abstract:Pre-training language models (LMs) on large-scale unlabeled text data makes the model much easier to achieve exceptional downstream performance than their counterparts directly trained on the downstream tasks. In this work, we study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks. We propose to use artificially constructed datasets as the pre-training data to exclude the effect of semantics, and further control what characteristics the pre-training corpora have. By fine-tuning the pre-trained models on GLUE benchmark, we can learn how beneficial it is to transfer the knowledge from the model trained on the dataset possessing that specific trait. We define and discuss three different characteristics in the artificial dataset: 1) matching the token's uni-gram or bi-gram distribution between pre-training and downstream fine-tuning, 2) the presence of the explicit dependencies among the tokens in a sequence, 3) the length of the implicit dependencies among the tokens in a sequence. Our experiments show that the explicit dependencies in the sequences of the pre-training data are critical to the downstream performance. Our results also reveal that models achieve better downstream performance when pre-trained on a dataset with a longer range of implicit dependencies. Based on our analysis, we find that models pre-trained with artificial datasets are prone to learn spurious correlation in downstream tasks. Our work reveals that even if the LMs are not pre-trained on natural language, they still gain transferability on certain human language downstream tasks once the LMs learn to model the token dependencies in the sequences. This result helps us understand the exceptional transferability of pre-trained LMs.

Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Models

Measuring Cross-lingual Transfer in Bytes

Data Contamination Can Cross Language Barriers

Investigating Data Contamination for Pre-training Language Models

From English To Foreign Languages: Transferring Pre-trained Language Models

On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets

Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models

Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Ability

Languages You Know Influence Those You Learn: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfer

mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models?

Investigating cross-lingual training for offensive language detection

Training Bilingual LMs with Data Constraints in the Targeted Language

GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost

Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning

Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale

How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning

Efficiently Adapting Pretrained Language Models To New Languages

Cross-lingual Transfer of Monolingual Models

Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models

Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation

An Efficient Approach for Studying Cross-Lingual Transfer in Multilingual Language Models