Abstract:Vision Transformer (ViT) acceleration with field programmable gate array (FPGA) is promising but challenging. Existing FPGA-based ViT accelerators mainly rely on temporal architectures, which process different operators by reusing the same hardware blocks and suffer from extensive memory access overhead. Pipelined architectures, either coarse-grained or fine-grained, unroll the ViT computation spatially for memory access efficiency. However, they usually suffer from significant hardware resource constraints and pipeline bubbles induced by the global computation dependency of ViT. In this paper, we introduce HG-PIPE, a pipelined FPGA accelerator for high-throughput and low-latency ViT processing. HG-PIPE features a hybrid-grained pipeline architecture to reduce on-chip buffer cost and couples the computation dataflow and parallelism design to eliminate the pipeline bubbles. HG-PIPE further introduces careful approximations to implement both linear and non-linear operators with abundant Lookup Tables (LUTs), thus alleviating resource constraints. On a ZCU102 FPGA, HG-PIPE achieves 2.78 times better throughput and 2.52 times better resource efficiency than the prior-art accelerators, e.g., AutoViTAcc. With a VCK190 FPGA, HG-PIPE realizes end-to-end ViT acceleration on a single device and achieves 7118 images/s, which is 2.81 times faster than a V100 GPU.

Research on application of pipeline technology based on FPGA

Research of high speed image processing system based on FPGA

Prototyping Design of a Flexible DSP Block with Pipeline Structure for FPGA

Design of A Fully Pipelined Single-Precision Floating-Point Unit

Research of High-Speed Pipelined Floating-Point Multipfier Design

A Pipeline Architecture for High Speed FIR Filters

High-Performance Reconfigurable Pipeline Implementation for FPGA-Based SmartNIC

Research and Implementation of Color Image Processing Pipeline Based on FPGA

An Automatic Mixed Software Hardware Pipeline Builder for CPU-FPGA Platforms

Design of Adder Based on Abacus Algorithm

FPGA Implementation of Feature Detection Algorithm Based on High Level Synthesis

A novel pipelined carry adder design based on half adder

HPSAP: A High-Performance and Synthesizable Asynchronous Pipeline With Quasi-2phase Conversion Method

An FPGA Based Parallel Pipelined FIR Filter Architecture

Implementation of Pipeline Structure of SM3 Algorithm on FPGA

HG-PIPE: Vision Transformer Acceleration with Hybrid-Grained Pipeline

Application Research of FPGA in Complex Logic System

Research on NetFPGA application technology

Framework for Application Mapping over Packet-Switched Network of FPGAs: Case Studies

Design of new pipeline arithmetic encoder

A Fast Approach For Generating Efficient Parsers On Fpgas