Abstract:Recent advancements in large language models (LLMs) boasting billions of parameters have generated a significant demand for efficient deployment in inference workloads. While hardware accelerators for Transformer-based models have been extensively studied, the majority of existing approaches rely on temporal architectures that reuse hardware units for different network layers and operators. However, these methods often encounter challenges in achieving low latency due to considerable memory access overhead. This paper investigates the feasibility and potential of model-specific spatial acceleration for LLM inference on FPGAs. Our approach involves the specialization of distinct hardware units for specific operators or layers, facilitating direct communication between them through a dataflow architecture while minimizing off-chip memory accesses. We introduce a comprehensive analytical model for estimating the performance of a spatial LLM accelerator, taking into account the on-chip compute and memory resources available on an FPGA. This model can be extended to multi-FPGA settings for distributed inference. Through our analysis, we can identify the most effective parallelization and buffering schemes for the accelerator and, crucially, determine the scenarios in which FPGA-based spatial acceleration can outperform its GPU-based counterpart. To enable more productive implementations of an LLM model on FPGAs, we further provide a library of high-level synthesis (HLS) kernels that are composable and reusable. This library will be made available as open-source. To validate the effectiveness of both our analytical model and HLS library, we have implemented BERT and GPT2 on an AMD Xilinx Alveo U280 FPGA device. Experimental results demonstrate our approach can achieve up to 13.4 × speedup when compared to previous FPGA-based accelerators for the BERT model. For GPT generative inference, we attain a 2.2 × speedup compared to DFX, an FPGA overlay, in the prefill stage, while achieving a 1.9 × speedup and a 5.7 × improvement in energy efficiency compared to the NVIDIA A100 GPU in the decode stage.

What problem does this paper attempt to address?

The paper primarily explores the potential of Field-Programmable Gate Arrays (FPGA) in the inference of Large Language Models (LLM), particularly focusing on FPGA-based spatial acceleration architectures. The core issue of the study is to evaluate whether FPGAs can effectively accelerate the inference process of LLMs and to explore their advantages over Graphics Processing Units (GPUs). Specifically, the paper addresses the following key issues: 1. **Analyzing the potential of FPGAs in LLM inference**: The authors evaluate the feasibility of FPGAs as accelerators by deeply analyzing the computational demands and memory access characteristics of Transformer-based LLMs in inference workloads. The study finds that traditional temporal architectures face high latency and energy consumption issues due to frequent off-chip memory accesses, thus proposing spatial architectures as a solution. 2. **Proposing a spatial acceleration architecture**: The paper introduces a specialized spatial architecture where different hardware units are designed to perform specific operations or layers, reducing off-chip memory accesses and improving data flow efficiency. This architecture minimizes memory access overhead through direct communication. 3. **Building an analytical model**: To estimate the performance of FPGA-based spatial accelerators, the paper establishes a comprehensive analytical model that considers the computational and memory resources on FPGAs. This model can be extended to multi-FPGA setups, supporting distributed inference. 4. **Experimental validation**: Through experimental validation, the authors demonstrate that their method achieves up to 13.4 times speedup on the BERT model compared to previous FPGA accelerators. For GPT generation inference, it achieves a 2.2 times speedup in the pre-fill stage compared to FPGA Dynamic Function eXchange (DFX) and a 1.9 times speedup and 5.7 times energy efficiency improvement in the decoding stage compared to the NVIDIA A100 GPU. In summary, the paper aims to achieve efficient and low-latency LLM inference acceleration through FPGAs and empirically proves the effectiveness of this approach.

Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference

Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference

A Comprehensive Evaluation of FPGA-Based Spatial Acceleration of LLMs

A Near Memory Computing FPGA Architecture for Neural Network Acceleration

LORA: A Latency-Oriented Recurrent Architecture for GPT Model on Multi-FPGA Platform with Communication Optimization

EdgeLLM: A Highly Efficient CPU-FPGA Heterogeneous Edge Accelerator for Large Language Models

Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective

The Feasibility of Implementing Large-Scale Transformers on Multi-FPGA Platforms

FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs

LlamaF: An Efficient Llama2 Architecture Accelerator on Embedded FPGAs

A Hardware Evaluation Framework for Large Language Model Inference

Research on LLM Acceleration Using the High-Performance RISC-V Processor "Xiangshan" (Nanhu Version) Based on the Open-Source Matrix Instruction Set Extension (Vector Dot Product)

HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level Synthesis

A Ubiquitous Machine Learning Accelerator With Automatic Parallelization on FPGA

FTRANS: Energy-Efficient Acceleration of Transformers using FPGA

Exploiting Intel Advanced Matrix Extensions (AMX) for Large Language Model Inference

Inference Performance Optimization for Large Language Models on CPUs

Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores

Task Scheduling for Efficient Inference of Large Language Models on Single Moderate GPU Systems

Enhancing Long Sequence Input Processing in FPGA-Based Transformer Accelerators Through Attention Fusion

Hardware Acceleration of LLMs: A comprehensive survey and comparison