Abstract:Multi-head self-attention (attention mechanism) has been employed in a variety of fields such as machine translation, language modeling, and image processing due to its superiority in feature extraction and sequential data analysis. This is benefited from a large number of parameters and sophisticated model architecture behind the attention mechanism. To efficiently deploy attention mechanism on resource-constrained devices, existing works propose to reduce the model size by building a customized smaller model or compressing a big standard model. A customized smaller model is usually optimized for the specific task and needs effort in model parameters exploration. Model compression reduces model size without hurting the model architecture robustness, which can be efficiently applied to different tasks. The compressed weights in the model are usually regularly shaped (e.g. rectangle) but the dimension sizes vary (e.g. differs in rectangle height and width). Such compressed attention mechanism can be efficiently deployed on CPU/GPU platforms as their memory and computing resources can be flexibly assigned with demand. However, for Field Programmable Gate Arrays (FPGAs), the data buffer allocation and computing kernel are fixed at run time to achieve maximum energy efficiency. After compression, weights are much smaller and different in size, which leads to inefficient utilization of FPGA on-chip buffer. Moreover, the different weight heights and widths may lead to inefficient FPGA computing kernel execution. Due to the large number of weights in the attention mechanism, building a unique buffer and computing kernel for each compressed weight on FPGA is not feasible. In this work, we jointly consider the compression impact on buffer allocation and the required computing kernel during the attention mechanism compressing. A novel structural pruning method with memory footprint awareness is proposed and the associated accelerator on FPGA is designed. The experimental results show that our work can compress Transformer (an attention mechanism based model) by 95x. The developed accelerator can fully utilize the FPGA resource, processing the sparse attention mechanism with the run-time throughput performance of 1.87 Tops in ZCU102 FPGA.

FuseMax: Leveraging Extended Einsums to Optimize Attention Accelerator Design

MAS-Attention: Memory-Aware Stream Processing for Attention Acceleration on Resource-Constrained Edge Devices

Enhancing Long Sequence Input Processing in FPGA-Based Transformer Accelerators Through Attention Fusion

FACT: FFN-Attention Co-optimized Transformer Architecture with Eager Correlation Prediction.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

FuseKNA: Fused Kernel Convolution based Accelerator for Deep Neural Networks

Hardware-Software Co-Design of an In-Memory Transformer Network Accelerator

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

Energon: Toward Efficient Acceleration of Transformers Using Dynamic Sparse Attention.

FAMOUS: Flexible Accelerator for the Attention Mechanism of Transformer on UltraScale+ FPGAs

An Analog and Digital Hybrid Attention Accelerator for Transformers with Charge-based In-memory Computing

Hyft: A Reconfigurable Softmax Accelerator with Hybrid Numeric Format for both Training and Inference

AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems

Ayaka: A Versatile Transformer Accelerator with Low-Rank Estimation and Heterogeneous Dataflow

Unified Accelerator for Attention and Convolution in Inference Based on FPGA

CoDA: A Co-Design Framework for Versatile and Efficient Attention Accelerators

A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipelining

Algorithm-hardware Co-design of Attention Mechanism on FPGA Devices

TURBOATTENTION: Efficient Attention Approximation For High Throughputs LLMs

FAST: Factorizable Attention for Speeding up Transformers

Balance Multi-Head Attention Based on Software and Hardware Co-design