Abstract:Recent advancements in large language models (LLMs) boasting billions of parameters have generated a significant demand for efficient deployment in inference workloads. While hardware accelerators for Transformer-based models have been extensively studied, the majority of existing approaches rely on temporal architectures that reuse hardware units for different network layers and operators. However, these methods often encounter challenges in achieving low latency due to considerable memory access overhead. This paper investigates the feasibility and potential of model-specific spatial acceleration for LLM inference on FPGAs. Our approach involves the specialization of distinct hardware units for specific operators or layers, facilitating direct communication between them through a dataflow architecture while minimizing off-chip memory accesses. We introduce a comprehensive analytical model for estimating the performance of a spatial LLM accelerator, taking into account the on-chip compute and memory resources available on an FPGA. This model can be extended to multi-FPGA settings for distributed inference. Through our analysis, we can identify the most effective parallelization and buffering schemes for the accelerator and, crucially, determine the scenarios in which FPGA-based spatial acceleration can outperform its GPU-based counterpart. To enable more productive implementations of an LLM model on FPGAs, we further provide a library of high-level synthesis (HLS) kernels that are composable and reusable. This library will be made available as open-source. To validate the effectiveness of both our analytical model and HLS library, we have implemented BERT and GPT2 on an AMD Xilinx Alveo U280 FPGA device. Experimental results demonstrate our approach can achieve up to 13.4 × speedup when compared to previous FPGA-based accelerators for the BERT model. For GPT generative inference, we attain a 2.2 × speedup compared to DFX, an FPGA overlay, in the prefill stage, while achieving a 1.9 × speedup and a 5.7 × improvement in energy efficiency compared to the NVIDIA A100 GPU in the decode stage.

Accelerating NLP Tasks on FPGA with Compressed BERT and a Hardware-Oriented Early Exit Method.

Hardware Acceleration of Fully Quantized BERT for Efficient Natural Language Processing

G-Bert: Enabling Green BERT Deployment on FPGA Via Hardware-Aware Hybrid Pruning

Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference

A Comprehensive Evaluation of FPGA-Based Spatial Acceleration of LLMs

Enhancing Long Sequence Input Processing in FPGA-Based Transformer Accelerators Through Attention Fusion

Accommodating Transformer Onto FPGA

FTRANS: Energy-Efficient Acceleration of Transformers using FPGA

Accelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimization

Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning

A Reconfigurable DNN Training Accelerator on FPGA

A High Energy-Efficiency FPGA-Based LSTM Accelerator Architecture Design by Structured Pruning and Normalized Linear Quantization

HPTA: A High Performance Transformer Accelerator Based on FPGA

An Automatic and Efficient BERT Pruning for Edge AI Systems

TransFRU: Efficient Deployment of Transformers on FPGA with Full Resource Utilization

Energy-efficient Task Adaptation for NLP Edge Inference Leveraging Heterogeneous Memory Architectures

Invited: Algorithm-Software-Hardware Co-Design for Deep Learning Acceleration

Improving Transformer Inference Through Optimized Non-Linear Operations with Quantization-Approximation-Based Strategy

Hardware Accelerator for High Accuracy Sign Language Recognition with Residual Network Based on FPGAs

EdgeLLM: A Highly Efficient CPU-FPGA Heterogeneous Edge Accelerator for Large Language Models