Abstract:Since Google proposed Transformer in 2017, it has made significant natural language processing (NLP) development. However, the increasing cost is a large amount of calculation and parameters. Previous researchers designed and proposed some accelerator structures for transformer models in field-programmable gate array (FPGA) to deal with NLP tasks efficiently. Now, the development of Transformer has also affected computer vision (CV) and has rapidly surpassed convolution neural networks (CNNs) in various image tasks. And there are apparent differences between the image data used in CV and the sequence data in NLP. The details in the models contained with transformer units in these two fields are also different. The difference in terms of data brings about the problem of the locality. The difference in the model structure brings about the problem of path dependence, which is not noticed in the existing related accelerator design. Therefore, in this work, we propose the ViA, a novel vision transformer (ViT) accelerator architecture based on FPGA, to execute the transformer application efficiently and avoid the cost of these challenges. By analyzing the data structure in the ViT, we design an appropriate partition strategy to reduce the impact of data locality in the image and improve the efficiency of computation and memory access. Meanwhile, by observing the computing flow of the ViT, we use the half-layer mapping and throughput analysis to reduce the impact of path dependence caused by the shortcut mechanism and fully utilize hardware resources to execute the Transformer efficiently. Based on optimization strategies, we design two reuse processing engines with the internal stream, different from the previous overlap or stream design patterns. In the stage of the experiment, we implement the ViA architecture in Xilinx Alveo U50 FPGA and finally achieved similar to 5.2 times improvement of energy efficiency compared with NVIDIA Tesla V100, and 4-10 times improvement of performance compared with related accelerators based on FPGA, that obtained nearly 309.6 GOP/s computing performance in the peek.

A Bandwidth Enhancement Method of VTA Based on Paralleled Memory Access Design

A Highly Configurable Hardware/Software Stack for DNN Inference Acceleration

A new VLSI design for Viterbi decoder based on ASIP

Leveraging the VTA-TVM Hardware-Software Stack for FPGA Acceleration of 8-bit ResNet-18 Inference

A 109-Gops/w FPGA-based Vision Transformer Accelerator with Weight-Loop Dataflow Featuring Data Reusing and Resource Saving

HSViT: A Hardware and Software Collaborative Design for Vision Transformer Via Multi-level Compression

Hardware-Aware Quantization and Performance Evaluation for Tensor Accelerator VTA

ME-ViT: A Single-Load Memory-Efficient FPGA Accelerator for Vision Transformers

A Novel Fully Hardware-Implemented SVD Solver Based on Ultra-Parallel BCV Jacobi Algorithm

A Hardware-Software Blueprint for Flexible Deep Learning Specialization

ViA: A Novel Vision-Transformer Accelerator Based on FPGA

Case Study: Optimization Methods With TVM Hybrid-OP on RISC-V Packed SIMD

Optimizing Cache Accesses with Tensor Memory Format Search for Transformers in TVM.

A highly parallel VLSI implementation for 8×8 integer transform in H.264/AVC

A Multiplier-less Transform Architecture with the Diagonal Data Mapping Transpose Memory for the AVS3 Standard

A Parallel Memory Architecture for Video Coding

TVM: An Automated End-to-End Optimizing Compiler for Deep Learning

Near-Memory Parallel Indexing and Coalescing: Enabling Highly Efficient Indirect Access for SpMV

Register Length Analysis and Vlsi Optimization of Vbs Hadamard Transform in H.264/Avc

Parallelizing and Optimizing H.264 on Synchronous Data Triggered Architecture

VAQF: Fully Automatic Software-hardware Co-design Framework for Low-bit Vision Transformer