Abstract: Pretrained language models (PLMs) such as BERT adopt a training paradigm which first pretrain the model in general data and then finetune the model on task-specific data, and have recently achieved great success. However, PLMs are notorious for their enormous parameters and hard to be deployed on real-life applications. Knowledge distillation has been prevailing to address this problem by transferring knowledge from a large teacher to a much smaller student over a set of data. We argue that the selection of thee three key components, namely teacher, training data, and learning objective, is crucial to the effectiveness of distillation. We, therefore, propose a four-stage progressive distillation framework ERNIE-Tiny to compress PLM, which varies the three components gradually from general level to task-specific level. Specifically, the first stage, General Distillation, performs distillation with guidance from pretrained teacher, gerenal data and latent distillation loss. Then, General-Enhanced Distillation changes teacher model from pretrained teacher to finetuned teacher. After that, Task-Adaptive Distillation shifts training data from general data to task-specific data. In the end, Task-Specific Distillation, adds two additional losses, namely Soft-Label and Hard-Label loss onto the last stage. Empirical results demonstrate the effectiveness of our framework and generalization gain brought by ERNIE-Tiny.In particular, experiments show that a 4-layer ERNIE-Tiny maintains over 98.0%performance of its 12-layer teacher BERT base on GLUE benchmark, surpassing state-of-the-art (SOTA) by 1.0% GLUE score with the same amount of parameters. Moreover, ERNIE-Tiny achieves a new compression SOTA on five Chinese NLP tasks, outperforming BERT base by 0.4% accuracy with 7.5x fewer parameters and9.4x faster inference speed.

Joint Dual Feature Distillation and Gradient Progressive Pruning for BERT compression

Joint Structured Pruning and Dense Knowledge Distillation for Efficient Transformer Model Compression

DDK: Dynamic structure pruning based on differentiable search and recursive knowledge distillation for BERT

Gradient-based Intra-attention Pruning on Pre-trained Language Models

Pruning before Fine-tuning: A Retraining-free Compression Framework for Pre-trained Language Models

Uncertainty-Driven Knowledge Distillation for Language Model Compression.

Deep Neural Compression Via Concurrent Pruning and Self-Distillation

LAD: Layer-Wise Adaptive Distillation for BERT Model Compression

Model Compression by Iterative Pruning with Knowledge Distillation and Its Application to Speech Enhancement

Contrastive Distillation on Intermediate Representations for Language Model Compression

EPSD: Early Pruning with Self-Distillation for Efficient Model Compression

Length-Adaptive Distillation: Customizing Small Language Model for Dynamic Token Pruning.

From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression

ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression

Patient Knowledge Distillation for BERT Model Compression

Explanation Guided Knowledge Distillation for Pre-trained Language Model Compression

Prune Once for All: Sparse Pre-Trained Language Models

Towards Efficient Pre-Trained Language Model Via Feature Correlation Distillation

Comb, Prune, Distill: Towards Unified Pruning for Vision Model Compression

Pruning-and-distillation: One-stage Joint Compression Framework for CNNs Via Clustering

Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding