Abstract:Since Google proposed Transformer in 2017, it has made significant natural language processing (NLP) development. However, the increasing cost is a large amount of calculation and parameters. Previous researchers designed and proposed some accelerator structures for transformer models in field-programmable gate array (FPGA) to deal with NLP tasks efficiently. Now, the development of Transformer has also affected computer vision (CV) and has rapidly surpassed convolution neural networks (CNNs) in various image tasks. And there are apparent differences between the image data used in CV and the sequence data in NLP. The details in the models contained with transformer units in these two fields are also different. The difference in terms of data brings about the problem of the locality. The difference in the model structure brings about the problem of path dependence, which is not noticed in the existing related accelerator design. Therefore, in this work, we propose the ViA, a novel vision transformer (ViT) accelerator architecture based on FPGA, to execute the transformer application efficiently and avoid the cost of these challenges. By analyzing the data structure in the ViT, we design an appropriate partition strategy to reduce the impact of data locality in the image and improve the efficiency of computation and memory access. Meanwhile, by observing the computing flow of the ViT, we use the half-layer mapping and throughput analysis to reduce the impact of path dependence caused by the shortcut mechanism and fully utilize hardware resources to execute the Transformer efficiently. Based on optimization strategies, we design two reuse processing engines with the internal stream, different from the previous overlap or stream design patterns. In the stage of the experiment, we implement the ViA architecture in Xilinx Alveo U50 FPGA and finally achieved similar to 5.2 times improvement of energy efficiency compared with NVIDIA Tesla V100, and 4-10 times improvement of performance compared with related accelerators based on FPGA, that obtained nearly 309.6 GOP/s computing performance in the peek.

An FPGA-Based Reconfigurable Accelerator for Convolution-Transformer Hybrid EfficientViT

ViA: A Novel Vision-Transformer Accelerator Based on FPGA

ME-ViT: A Single-Load Memory-Efficient FPGA Accelerator for Vision Transformers

CHOSEN: Compilation to Hardware Optimization Stack for Efficient Vision Transformer Inference

FPGA-Based Vit Inference Accelerator Optimization

Auto-ViT-Acc: an FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization

ViTA: A Vision Transformer Inference Accelerator for Edge Applications

HG-PIPE: Vision Transformer Acceleration with Hybrid-Grained Pipeline

Unified Accelerator for Attention and Convolution in Inference Based on FPGA

Trio-ViT: Post-Training Quantization and Acceleration for Softmax-Free Efficient Vision Transformer

M$^2$-ViT: Accelerating Hybrid Vision Transformers with Two-Level Mixed Quantization

Resource-Efficient Optimization for FPGA-Based Convolution Accelerator

FET-OPU: A Flexible and Efficient FPGA-Based Overlay Processor for Transformer Networks

TransFRU: Efficient Deployment of Transformers on FPGA with Full Resource Utilization

An Efficient FPGA-based Accelerator Design for Convolution

Accommodating Transformer Onto FPGA

A High Performance Reconfigurable Hardware Architecture for Lightweight Convolutional Neural Network

FTRANS: Energy-Efficient Acceleration of Transformers using FPGA

Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers

A High-Performance Accelerator for Large-Scale Convolutional Neural Networks

A Flexible and Efficient FPGA Accelerator for Various Large-Scale and Lightweight CNNs