Abstract:Resistive random-access-memory (ReRAM) crossbar is a promising technique for deep neural network (DNN) accelerators, thanks to its in-memory and in-situ analog computing abilities for vector–matrix multiplication-and-accumulations (VMMs). However, it is challenging for crossbar architecture to exploit the sparsity in DNNs. It is inevitably complex and costly to exploit fine-grained sparsity due to the limitation of the tightly coupled crossbar structure. As a countermeasure, we develop a novel ReRAM-based DNN accelerator, named sparse-multiplication-engine (SME), based on a hardware and software co-design framework. First, we orchestrate the bit-sparse pattern to increase the density of bit-sparsity based on existing quantization methods. Such quantized weights can be nicely generated using the alternating direction method of multipliers (ADMM) optimization during the DNN fine-tuning, which can exactly enforce bit patterns in weights. Second, we propose a novel weight mapping mechanism to slice the bits of the weight across crossbars and splice the activation results in peripheral circuits. This mechanism can decouple the tightly coupled crossbar structure and cumulate the sparsity in the crossbar. Finally, a superior squeeze-out scheme empties the crossbars mapped with highly sparse nonzeros from the previous two steps. We design the SME architecture and discuss its use for other quantization methods and different ReRAM cell technologies. We further propose a workload grouping algorithm and a pipeline to achieve workload balance among crossbar-rows that concurrently execute multiply–accumulate operations to optimize the system latency. Putting all together, with the optimized model, compared with prior state-of-the-art designs, the SME shrinks the use of crossbars up to $8.7\times $ and $2.1\times $ using ResNet-50 and MobileNet-v2, respectively, and achieve average $3.1\times $ speed up with no or little accuracy loss on ImageNet.

VerSA: Versatile Systolic Array Architecture for Sparse and Dense Matrix Multiplications

Sparse Systolic Tensor Array for Efficient CNN Hardware Acceleration

DeMM: A Decoupled Matrix Multiplication Engine Supporting Relaxed Structured Sparsity

ONE-SA: Enabling Nonlinear Operations in Systolic Arrays for Efficient and Flexible Neural Network Inference

Dedicated Hardware Accelerators for Processing of Sparse Matrices and Vectors: A Survey

ReSA: Reconfigurable Systolic Array for Multiple Tiny DNN Tensors

Diagonally-Addressed Matrix Nicknack: How to improve SpMV performance

SoBS-X: Squeeze-Out Bit Sparsity for ReRAM-Crossbar-Based Neural Network Accelerator.

FlexSA: Flexible Systolic Array Architecture for Efficient Pruned DNN Model Training

Configurable Multi-directional Systolic Array Architecture for Convolutional Neural Networks

CONNA: Configurable Matrix Multiplication Engine for Neural Network Acceleration

Cerberus: Triple Mode Acceleration of Sparse Matrix and Vector Multiplication

A Data Locality-Aware Design Framework For Reconfigurable Sparse Matrix-Vector Multiplication Kernel

dCSR: A Memory-Efficient Sparse Matrix Representation for Parallel Neural Network Inference

Sense: Model-Hardware Codesign for Accelerating Sparse CNNs on Systolic Arrays

Systolic Array Data Flows for Efficient Matrix Multiplication in Deep Neural Networks

A Systolic Array-Based Scheduling Strategy for Sparse CNN Accelerators

Sparse Matrix Multiplication On An Associative Processor

Esspmv: an Embedded-FPGA-based Hardware Accelerator for Symmetric Sparse Matrix-Vector Multiplication.

Sense: Model Hardware Co-design for Accelerating Sparse CNN on Systolic Array