Abstract: Despite the remarkable success of pre-trained language models (PLMs), they still face two challenges: First, large-scale PLMs are inefficient in terms of memory footprint and computation. Second, on the downstream tasks, PLMs tend to rely on the dataset bias and struggle to generalize to out-of-distribution (OOD) data. In response to the efficiency problem, recent studies show that dense PLMs can be replaced with sparse subnetworks without hurting the performance. Such subnetworks can be found in three scenarios: 1) the fine-tuned PLMs, 2) the raw PLMs and then fine-tuned in isolation, and even inside 3) PLMs without any parameter fine-tuning. However, these results are only obtained in the in-distribution (ID) setting. In this paper, we extend the study on PLMs subnetworks to the OOD setting, investigating whether sparsity and robustness to dataset bias can be achieved simultaneously. To this end, we conduct extensive experiments with the pre-trained BERT model on three natural language understanding (NLU) tasks. Our results demonstrate that \textbf{sparse and robust subnetworks (SRNets) can consistently be found in BERT}, across the aforementioned three scenarios, using different training and compression methods. Furthermore, we explore the upper bound of SRNets using the OOD information and show that \textbf{there exist sparse and almost unbiased BERT subnetworks}. Finally, we present 1) an analytical study that provides insights on how to promote the efficiency of SRNets searching process and 2) a solution to improve subnetworks' performance at high sparsity. The code is available at https://github.com/llyx97/sparse-and-robust-PLM.

RoBERTa: A Robustly Optimized BERT Pretraining Approach

A Robustly Optimized BERT Pre-training Approach with Post-training

bert2BERT: Towards Reusable Pretrained Language Models

ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT

MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

EarlyBERT: Efficient BERT Training Via Early-bird Lottery Tickets

Breaking MLPerf Training: A Case Study on Optimizing BERT

A Win-win Deal: Towards Sparse and Robust Pre-trained Language Models

Visualizing and Understanding the Effectiveness of BERT.

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

{E}fficient{BERT}: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation

StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding

DACBERT: Leveraging Dependency Agreement for Cost-Efficient Bert Pretraining

Enhancing Scalability of Pre-trained Language Models Via Efficient Parameter Sharing.

Efficient Fine-Tuning of Compressed Language Models with Learners

Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)

Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting

RoChBert: Towards Robust BERT Fine-tuning for Chinese

NarrowBERT: Accelerating Masked Language Model Pretraining and Inference