Abstract:The attention mechanism is becoming a vital building block across various modern neural networks, e.g., Transformers. However, it encounters low efficiency when deployed on the general-purpose GPU/CPU platform, which motivates the dedicated accelerator design. Existing accelerators are commonly devised by exploring the potential sparsity in attention mechanism using a hardware-software co-design scheme, which suffers from complicated training, fine-tuning processes, and possible accuracy degradation. More importantly, the sparse pattern only focuses on certain datasets with less generality, and the fine-grained sparse pattern could also bring hardware inefficiency. Instead, we try to solve these issues from another perspective: By systematically analyzing the inherent dataflow characteristics of the attention mechanism, we propose the Co-Operative Systolic Arrays (COSA) with an optimized dataflow to support general purpose attention mechanism and pursue higher computational efficiency. COSA system exploits the high parallelism from the inherent model and leverages run-time configurable hybrid dataflows, i.e., weight and output stationary for a systolic array to support the varying matrix multiplication in the attention mechanism. Regarding the cascaded matrix multiplications, COSA proposes levels of fusion methodologies to reduce off-chip access and enhance PE utilization, such as directly using the result of output stationary as the weight of weight stationary systolic array by deep fusion. Additionally, COSA system also provides the solution to hide the latency and radically save the buffer size related to softmax. Experiment results show that, across various benchmarks, COSA can achieve 2.29-2.60× throughput improvement over the traditional systolic array of the same MAC number, with up to 94.7% PE utilization rate and 8.2× less off-chip memory access. Compared with general-purpose platforms, 7.6-12.4× energy efficiency over NVIDIA GeForce 3090 GPU and 35.2-80.9× energy efficiency over Intel 6226R server CPU.

Configurable Multi-directional Systolic Array Architecture for Convolutional Neural Networks

A High-Performance Systolic Array Accelerator Dedicated for CNN.

CSA-CiM: Enhancing Multi-Functional Computing-in-Memory with Configurable Sense Amplifiers

On the Difficulty of Designing Processor Arrays for Deep Neural Networks

ReSA: Reconfigurable Systolic Array for Multiple Tiny DNN Tensors

ReDas: A Lightweight Architecture for Supporting Fine-Grained Reshaping and Multiple Dataflows on Systolic Array

ONE-SA: Enabling Nonlinear Operations in Systolic Arrays for Efficient and Flexible Neural Network Inference

COSY: an Energy-Efficient Hardware Architecture for Deep Convolutional Neural Networks Based on Systolic Array.

Sense: Model Hardware Co-design for Accelerating Sparse CNN on Systolic Array

A Scalable 3D Array Architecture for Accelerating Convolutional Neural Networks

Sense: Model-Hardware Codesign for Accelerating Sparse CNNs on Systolic Arrays

Generating Systolic Array Accelerators with Reusable Blocks

COSA Plus: Enhanced Co-Operative Systolic Arrays for Attention Mechanism in Transformers

Accelerating Convolutional Neural Network Inference Based on a Reconfigurable Sliced Systolic Array

A Systolic Computing-in-Memory Array Based Accelerator with Predictive Early Activation for Spatiotemporal Convolutions

ArrayFlex: A Systolic Array Architecture with Configurable Transparent Pipelining

An Energy-Efficient Convolutional Neural Network Processor Architecture Based on a Systolic Array

DSA-CNN: an fpga-integrated deformable systolic array for convolutional neural network acceleration

VerSA: Versatile Systolic Array Architecture for Sparse and Dense Matrix Multiplications

S2Engine: A Novel Systolic Architecture for Sparse Convolutional Neural Networks

The Storage Structure of Convolutional Neural Network Reconfigurable Accelerator Based on ASIC