Abstract:With the yearning for deep learning democratization, there are increasing demands to implement Transformer-based natural language processing (NLP) models on resource-constrained devices for low-latency and high accuracy. Existing BERT pruning methods require domain experts to heuristically handcraft hyperparameters to strike a balance among model size, latency, and accuracy. In this work, we propose AE-BERT, an automatic and efficient BERT pruning framework with efficient evaluation to select a "good" sub-network candidate (with high accuracy) given the overall pruning ratio constraints. Our proposed method requires no human experts experience and achieves a better accuracy performance on many NLP tasks. Our experimental results on General Language Understanding Evaluation (GLUE) benchmark show that AE-BERT outperforms the state-of-the-art (SOTA) hand-crafted pruning methods on BERT$_{\mathrm{BASE}}$. On QNLI and RTE, we obtain 75\% and 42.8\% more overall pruning ratio while achieving higher accuracy. On MRPC, we obtain a 4.6 higher score than the SOTA at the same overall pruning ratio of 0.5. On STS-B, we can achieve a 40\% higher pruning ratio with a very small loss in Spearman correlation compared to SOTA hand-crafted pruning methods. Experimental results also show that after model compression, the inference time of a single BERT$_{\mathrm{BASE}}$ encoder on Xilinx Alveo U200 FPGA board has a 1.83$\times$ speedup compared to Intel(R) Xeon(R) Gold 5218 (2.30GHz) CPU, which shows the reasonableness of deploying the proposed method generated subnets of BERT$_{\mathrm{BASE}}$ model on computation restricted devices.

Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning

Joint Structured Pruning and Dense Knowledge Distillation for Efficient Transformer Model Compression

Prune Once for All: Sparse Pre-Trained Language Models

Structured Pruning of Large Language Models

An Automatic and Efficient BERT Pruning for Edge AI Systems

BlockPruner: Fine-grained Pruning for Large Language Models

FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing

Aggressive Post-Training Compression on Extremely Large Language Models

Large Language Models Compression via Low-Rank Feature Distillation

Can pruning make Large Language Models more efficient?

Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations

G-Bert: Enabling Green BERT Deployment on FPGA Via Hardware-Aware Hybrid Pruning

Gradient-Free Structured Pruning with Unlabeled Data

Structured Pruning Learns Compact and Accurate Models

Reconstruct the Pruned Model without Any Retraining

Block Pruning For Faster Transformers

Block-wise Bit-Compression of Transformer-based Models

Fluctuation-based Adaptive Structured Pruning for Large Language Models

LLM-BIP: Structured Pruning for Large Language Models with Block-Wise Forward Importance Propagation

Structured Pruning of a BERT-based Question Answering Model

LLM-Pruner: On the Structural Pruning of Large Language Models