Abstract:Models based on the attention mechanism, i.e., transformers, have shown extraordinary performance in natural language processing (NLP) tasks. However, their memory footprint, inference latency, and power consumption are still prohibitive for efficient inference at edge devices, even at data centers. To tackle this issue, we present an algorithm-architecture co-design named DTATrans. We find empirically that the tolerance to the noise varies from token to token in attention-based NLP models. This finding leads us to dynamically quantize different tokens with mixed levels of bits. Furthermore, we find that the overstrict quantization method causes a dilemma of the model accuracy and model compression ratio, which impels us to explore a method to compensate for the model accuracy when the compression ratio is high. Thus, in DTATrans, we design a compression framework that: 1) dynamically quantizes tokens while they are forwarded in the models; 2) jointly determines the ratio of each precision; and 3) compensate the model accuracy by exploiting lightweight computing on the 0-bit tokens. Moreover, due to the dynamic mixed-precision tokens caused by our framework, previous matrix-multiplication accelerators (e.g., systolic array) cannot effectively exploit the benefit of the compressed attention computation. We thus design our transformer accelerator with the variable-speed systolic array (VSSA) and propose an effective optimization strategy to alleviate the pipeline-stall problem in VSSA without hardware overhead. We conduct experiments with existing attention-based NLP models, including BERT and GPT-2 on various language tasks. Our results show that DTATrans outperforms the previous neural network accelerator Eyeriss by 16.04x in terms of speedup and 3.62x in terms of energy saving. Compared with the state-ofthe-art attention accelerator SpAtten, our DTATrans achieves at least 3.62x speedup and 4.22x energy efficiency improvement.

Fitop-Trans: Maximizing Transformer Pipeline Efficiency Through Fixed-Length Token Pruning on FPGA

PP-Transformer: Enable Efficient Deployment of Transformers Through Pattern Pruning

A Fast Post-Training Pruning Framework for Transformers

Accommodating Transformer Onto FPGA

TransFRU: Efficient Deployment of Transformers on FPGA with Full Resource Utilization

FTRANS: Energy-Efficient Acceleration of Transformers using FPGA

A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipelining

Enhancing Long Sequence Input Processing in FPGA-Based Transformer Accelerators Through Attention Fusion

LTrans-OPU: A Low-Latency FPGA-Based Overlay Processor for Transformer Networks

Accelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimization

Constraint-aware and Ranking-distilled Token Pruning for Efficient Transformer Inference

An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers

DTATrans: Leveraging Dynamic Token-Based Quantization with Accuracy Compensation Mechanism for Efficient Transformer Architecture.

Learned Token Pruning for Transformers

SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers

FET-OPU: A Flexible and Efficient FPGA-Based Overlay Processor for Transformer Networks

A Runtime-Adaptive Transformer Neural Network Accelerator on FPGAs

HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level Synthesis

Improving Transformer Inference Through Optimized Non-Linear Operations with Quantization-Approximation-Based Strategy

ProTEA: Programmable Transformer Encoder Acceleration on FPGA

A Cost-Efficient FPGA Implementation of Tiny Transformer Model using Neural ODE