Abstract:Since Google proposed Transformer in 2017, it has made significant natural language processing (NLP) development. However, the increasing cost is a large amount of calculation and parameters. Previous researchers designed and proposed some accelerator structures for transformer models in field-programmable gate array (FPGA) to deal with NLP tasks efficiently. Now, the development of Transformer has also affected computer vision (CV) and has rapidly surpassed convolution neural networks (CNNs) in various image tasks. And there are apparent differences between the image data used in CV and the sequence data in NLP. The details in the models contained with transformer units in these two fields are also different. The difference in terms of data brings about the problem of the locality. The difference in the model structure brings about the problem of path dependence, which is not noticed in the existing related accelerator design. Therefore, in this work, we propose the ViA, a novel vision transformer (ViT) accelerator architecture based on FPGA, to execute the transformer application efficiently and avoid the cost of these challenges. By analyzing the data structure in the ViT, we design an appropriate partition strategy to reduce the impact of data locality in the image and improve the efficiency of computation and memory access. Meanwhile, by observing the computing flow of the ViT, we use the half-layer mapping and throughput analysis to reduce the impact of path dependence caused by the shortcut mechanism and fully utilize hardware resources to execute the Transformer efficiently. Based on optimization strategies, we design two reuse processing engines with the internal stream, different from the previous overlap or stream design patterns. In the stage of the experiment, we implement the ViA architecture in Xilinx Alveo U50 FPGA and finally achieved similar to 5.2 times improvement of energy efficiency compared with NVIDIA Tesla V100, and 4-10 times improvement of performance compared with related accelerators based on FPGA, that obtained nearly 309.6 GOP/s computing performance in the peek.

SWAT: an Efficient Swin Transformer Accelerator Based on FPGA

An Efficient FPGA-Based Accelerator for Swin Transformer

Swin-Free: Achieving Better Cross-Window Attention and Efficiency with Size-varying Window

MSTRIQ: No Reference Image Quality Assessment Based on Swin Transformer with Multi-Stage Fusion

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

TransFRU: Efficient Deployment of Transformers on FPGA with Full Resource Utilization

Swin Transformer with Local Aggregation

ViA: A Novel Vision-Transformer Accelerator Based on FPGA

SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection

An efficient swin transformer-based method for underwater image enhancement

SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers

An FPGA-Based Reconfigurable Accelerator for Convolution-Transformer Hybrid EfficientViT

SwinFace: A Multi-task Transformer for Face Recognition, Expression Recognition, Age Estimation and Attribute Estimation

VAQF: Fully Automatic Software-hardware Co-design Framework for Low-bit Vision Transformer

HPTA: A High Performance Transformer Accelerator Based on FPGA

A Cost-Efficient FPGA Implementation of Tiny Transformer Model using Neural ODE

Unified Accelerator for Attention and Convolution in Inference Based on FPGA

Swin Transformer for Fast MRI

Accelerating Attention Mechanism on FPGAs Based on Efficient Reconfigurable Systolic Array