Abstract:The industrial usage of huge pre-training language models such as BERT and ALBERT are limited by the computational probability problem in the fine-tuning process and overthinking problem in the inference process. PF-BERxiT has been proposed to optimize the pre-trained languages with a novel parameter-efficient fine-tuning method and a flexible early exiting strategy. Significantly, the new parameter-efficient fine-tuning model integrates a bottleneck adapter architecture parallel to the transformer architecture, and only the adapter's parameters are adjusted. In addition, we integrate an extra sub-learning module to learn the samples' characteristics, improving the accuracy and efficiency simultaneously. The flexible exiting strategy allows the model to exit early if the similarity score of adjacent layers is less than the threshold for pre-defined times. It is more flexible than previous early exiting methods, for it can simultaneously adjust the similarity score thresholds and patience parameters according to the request traffic. Extensive experiments are conducted on the GLUE benchmark, demonstrating that: (1) PF-BERxiT outperforms conventional training and parameter-efficient strategies with only a few parameters fine-tuned. (2) PF-BERxiT strikes a better balance between model performances and speedup ratios than previous state-of-the-art (SOTA) early exiting methods such as PABEE and BERxiT. (3) Ablation studies in the fine-tuning process demonstrate that the best bottleneck dimension r of the adapters is 32, and the adapters placed parallel to the feed-forward module are more efficient. (4) Ablation studies in the inference process demonstrate that for variants of PF-BERxiT with different similarity scores, PFBERxiT-kl and PF-BERxiT-bikl attain better speedup-accuracy trade-offs than PF-BERxiT-rekl. Our PF-BERxiT helps attain a better trade-off between performance and efficiency, providing a reference for the efficient application of neural computing.

Accelerating BERT inference with GPU-efficient exit prediction

SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference.

PCEE-BERT: Accelerating BERT Inference Via Patient and Confident Early Exiting

FastBERT: a Self-distilling BERT with Adaptive Inference Time

Exploiting Student Parallelism for Low-latency GPU Inference of BERT-like Models in Online Services

BADGE: Speeding Up BERT Inference after Deployment Via Block-wise Bypasses and Divergence-based Early Exiting.

FastNER: Speeding up Inferences for Named Entity Recognition Tasks.

PF-BERxiT: Early Exiting for BERT with Parameter-Efficient Fine-Tuning and Flexible Early Exiting Strategy.

{E}fficient{BERT}: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation

Elbert: Fast Albert with Confidence-Window Based Early Exit

ME-BERT: Multi-exit BERT by Use of Adapter.

SkipBERT: Efficient Inference with Shallow Layer Skipping

Accelerating NLP Tasks on FPGA with Compressed BERT and a Hardware-Oriented Early Exit Method.

EarlyBERT: Efficient BERT Training Via Early-bird Lottery Tickets

DE$^3$-BERT: Distance-Enhanced Early Exiting for BERT based on Prototypical Networks

Dynamic and Efficient Inference for Text Generation Via BERT Family

Fast and Accurate FSA System Using ELBERT: An Efficient and Lightweight BERT

Exponentially Faster Language Modelling

LeeBERT: Learned Early Exit for BERT with Cross-Level Optimization

SlowBERT: Slow-down Attacks on Input-adaptive Multi-exit BERT

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training