Abstract:Adjusting the latency, power, and accuracy of natural language understanding models is a desirable objective of an efficient architecture. This paper proposes an efficient Transformer architecture that adjusts the inference computational cost adaptively with a desired inference latency speedup. In fine-tuning phase, the proposed method detects less important hidden sequence elements (word-vectors) and eliminates them in each encoder layer using a proposed Attention Context Contribution (ACC) metric. After the fine-tuning phase, with the novel offline-tuning property, the inference latency of the model can be adjusted in a wide range of inference speedup selections without any further training. The proposed method is applied to the BERT_base, GPT-2 and Flan-T5 models for evaluation. Extensive experiments show that most of the word-vectors in higher Transformer layers have less contribution to the subsequent layers; hence, they can be eliminated to improve the inference latency. Experimental results on extensive sentiment analysis, classification, text generation tasks and regression benchmarks like GLUE showed that the method is effective in various datasets with minimal impact on the input's global context. The method was also evaluated under the instruction tuning paradigm, and its performance was measured using different types of prompting. The proposed method mathematically and experimentally improves the inference latency of BERT_base and GPT-2 by up to 4.8 and 3.72 times with less than 0.75% accuracy drop and passable perplexity on average. The suggested approach posits that in Large Language Models (LLMs), although the complete network is necessary for training, it can be truncated during the fine-tuning phase.

LayerConnect: Hypernetwork-Assisted Inter-Layer Connector to Enhance Parameter Efficiency.

Enhancing Parameter Efficiency in Model Inference Using an Ultralight Inter-Transformer Linear Structure

Hadamard Adapter: An Extreme Parameter-Efficient Adapter Tuning Method for Pre-trained Language Models

COST-EFF: Collaborative Optimization of Spatial and Temporal Efficiency with Slenderized Multi-exit Language Models

Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models

Transformer Layer Injection: A Novel Approach for Efficient Upscaling of Large Language Models

Parameter-Efficient Transfer Learning for NLP

Parameter-efficient Tuning for Large Language Model Without Calculating Its Gradients

Latency Adjustable Transformer Encoder for Language Understanding

SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters

Predicting Rewards Alongside Tokens: Non-disruptive Parameter Insertion for Efficient Inference Intervention in Large Language Model

One Network, Many Masks: Towards More Parameter-Efficient Transfer Learning

Exploring Extreme Parameter Compression for Pre-trained Language Models

HyperLoader: Integrating Hypernetwork-Based LoRA and Adapter Layers into Multi-Task Transformers for Sequence Labelling

Parameter Efficient Multi-task Model Fusion with Partial Linearization

Parameter-Efficient Fine-Tuning With Adapters

A Multi-Level Framework for Accelerating Training Transformer Models

Smartformer: An Intelligent Model Compression Framework for Transformer

FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models

Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA

Dynamic Layer Tying for Parameter-Efficient Transformers