Abstract:Pre-training language models (LMs) on large-scale unlabeled text data makes the model much easier to achieve exceptional downstream performance than their counterparts directly trained on the downstream tasks. In this work, we study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks. We propose to use artificially constructed datasets as the pre-training data to exclude the effect of semantics, and further control what characteristics the pre-training corpora have. By fine-tuning the pre-trained models on GLUE benchmark, we can learn how beneficial it is to transfer the knowledge from the model trained on the dataset possessing that specific trait. We define and discuss three different characteristics in the artificial dataset: 1) matching the token's uni-gram or bi-gram distribution between pre-training and downstream fine-tuning, 2) the presence of the explicit dependencies among the tokens in a sequence, 3) the length of the implicit dependencies among the tokens in a sequence. Our experiments show that the explicit dependencies in the sequences of the pre-training data are critical to the downstream performance. Our results also reveal that models achieve better downstream performance when pre-trained on a dataset with a longer range of implicit dependencies. Based on our analysis, we find that models pre-trained with artificial datasets are prone to learn spurious correlation in downstream tasks. Our work reveals that even if the LMs are not pre-trained on natural language, they still gain transferability on certain human language downstream tasks once the LMs learn to model the token dependencies in the sequences. This result helps us understand the exceptional transferability of pre-trained LMs.

How Transferable Are Neural Networks in NLP Applications?

Exploring and Predicting Transferability across NLP Tasks

Transfer Learning and Deep Domain Adaptation

Towards Understanding the Transferability of Deep Representations

Towards Interpreting Recurrent Neural Networks Through Probabilistic Abstraction

Transferability in Deep Learning: A Survey

Rethinking Two Consensuses of the Transferability in Deep Learning

How transferable are features in deep neural networks?

Human-centric Transfer Learning Explanation Via Knowledge Graph [extended Abstract]

Deep Model Transferability from Attribution Maps

What is being transferred in transfer learning?

A Survey on Transferability of Adversarial Examples across Deep Neural Networks

On the Transferability of Adversarial Attacksagainst Neural Text Classifier

Understanding the Transferability of Representations via Task-Relatedness

Transferability of Neural Network Clinical De-identification Systems

Transferable Normalization: Towards Improving Transferability of Deep Neural Networks

Transfer Learning and Its Application Research

On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets

Is Robustness Transferable across Languages in Multilingual Neural Machine Translation?

Transferable Post-training via Inverse Value Learning

Neural Task Representations as Weak Supervision for Model Agnostic Cross-Lingual Transfer