Abstract:While pre-trained language models (e.g., BERT) have achieved impressive results on different natural language processing tasks, they have large numbers of parameters and suffer from big computational and memory costs, which make them difficult for real-world deployment. Therefore, model compression is necessary to reduce the computation and memory cost of pre-trained models. In this work, we aim to compress BERT and address the following two challenging practical issues: (1) The compression algorithm should be able to output multiple compressed models with different sizes and latencies, in order to support devices with different memory and latency limitations; (2) The algorithm should be downstream task agnostic, so that the compressed models are generally applicable for different downstream tasks. We leverage techniques in neural architecture search (NAS) and propose NAS-BERT, an efficient method for BERT compression. NAS-BERT trains a big supernet on a search space containing a variety of architectures and outputs multiple compressed models with adaptive sizes and latency. Furthermore, the training of NAS-BERT is conducted on standard self-supervised pre-training tasks (e.g., masked language model) and does not depend on specific downstream tasks. Thus, the compressed models can be used across various downstream tasks. The technical challenge of NAS-BERT is that training a big supernet on the pre-training task is extremely costly. We employ several techniques including block-wise search, search space pruning, and performance approximation to improve search efficiency and accuracy. Extensive experiments on GLUE and SQuAD benchmark datasets demonstrate that NAS-BERT can find lightweight models with better accuracy than previous approaches, and can be directly applied to different downstream tasks with adaptive model sizes for different requirements of memory or latency.

Optimizing small BERTs trained for German NER

Load What You Need: Smaller Versions of Multilingual BERT

GottBERT: a pure German Language Model

Efficient Fine-Tuning of BERT Models on the Edge

Symmetric Dot-Product Attention for Efficient Training of BERT Language Models

NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search

Developing and Evaluating Tiny to Medium-Sized Turkish BERT Models

Fine-tuning large neural language models for biomedical natural language processing

NarrowBERT: Accelerating Masked Language Model Pretraining and Inference

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

Low-Memory Neural Network Training: A Technical Report

The Right Tool for the Job: Matching Model and Instance Complexities

A More Efficient Chinese Named Entity Recognition base on BERT and Syntactic Analysis

Efficient Fine-Tuning of Compressed Language Models with Learners

EarlyBERT: Efficient BERT Training Via Early-bird Lottery Tickets

Exploring Cross-sentence Contexts for Named Entity Recognition with BERT

Towards Making the Most of BERT in Neural Machine Translation

Less is More! A slim architecture for optimal language translation

BERT might be Overkill: A Tiny but Effective Biomedical Entity Linker based on Residual Convolutional Neural Networks

Training dataset and dictionary sizes matter in BERT models: the case of Baltic languages

Extremely Small BERT Models from Mixed-Vocabulary Training