Abstract:The attention mechanism is becoming a vital building block across various modern neural networks, e.g., Transformers. However, it encounters low efficiency when deployed on the general-purpose GPU/CPU platform, which motivates the dedicated accelerator design. Existing accelerators are commonly devised by exploring the potential sparsity in attention mechanism using a hardware-software co-design scheme, which suffers from complicated training, fine-tuning processes, and possible accuracy degradation. More importantly, the sparse pattern only focuses on certain datasets with less generality, and the fine-grained sparse pattern could also bring hardware inefficiency. Instead, we try to solve these issues from another perspective: By systematically analyzing the inherent dataflow characteristics of the attention mechanism, we propose the Co-Operative Systolic Arrays (COSA) with an optimized dataflow to support general purpose attention mechanism and pursue higher computational efficiency. COSA system exploits the high parallelism from the inherent model and leverages run-time configurable hybrid dataflows, i.e., weight and output stationary for a systolic array to support the varying matrix multiplication in the attention mechanism. Regarding the cascaded matrix multiplications, COSA proposes levels of fusion methodologies to reduce off-chip access and enhance PE utilization, such as directly using the result of output stationary as the weight of weight stationary systolic array by deep fusion. Additionally, COSA system also provides the solution to hide the latency and radically save the buffer size related to softmax. Experiment results show that, across various benchmarks, COSA can achieve 2.29-2.60× throughput improvement over the traditional systolic array of the same MAC number, with up to 94.7% PE utilization rate and 8.2× less off-chip memory access. Compared with general-purpose platforms, 7.6-12.4× energy efficiency over NVIDIA GeForce 3090 GPU and 35.2-80.9× energy efficiency over Intel 6226R server CPU.

Heterogeneous Systolic Array Architecture for Compact CNNs Hardware Accelerators

EWS: an Energy-Efficient CNN Accelerator with Enhanced Weight Stationary Dataflow

A High-Performance Systolic Array Accelerator Dedicated for CNN.

Configurable Multi-directional Systolic Array Architecture for Convolutional Neural Networks

A Convolutional Neural Network Accelerator Architecture with Fine-Granular Mixed Precision Configurability.

Sense: Model Hardware Co-design for Accelerating Sparse CNN on Systolic Array

Sense: Model-Hardware Codesign for Accelerating Sparse CNNs on Systolic Arrays

A High-Efficient and Configurable Hardware Accelerator for Convolutional Neural Network

HSAS: Efficient task scheduling for large scale heterogeneous systolic array accelerator cluster

A Systolic Array-Based Scheduling Strategy for Sparse CNN Accelerators

Generating Systolic Array Accelerators with Reusable Blocks

Automated Systolic Array Architecture Synthesis for High Throughput CNN Inference on FPGAs.

VWA: Hardware Efficient Vectorwise Accelerator for Convolutional Neural Network

Addressing the issue of processing element under-utilization in general-purpose systolic deep learning accelerators

Relative Indexed Compressed Sparse Filter Encoding Format for Hardware-Oriented Acceleration of Deep Convolutional Neural Networks

An Efficient Streaming Accelerator for Low Bit-Width Convolutional Neural Networks

FSA: A Fine-Grained Systolic Accelerator for Sparse CNNs

On the Difficulty of Designing Processor Arrays for Deep Neural Networks

COSA Plus: Enhanced Co-Operative Systolic Arrays for Attention Mechanism in Transformers

Sparse Systolic Tensor Array for Efficient CNN Hardware Acceleration

An Energy-Efficient Convolutional Neural Network Processor Architecture Based on a Systolic Array