Abstract:Fine-grained sparse convolutional neural networks (CNNs) achieve a better trade-off between model accuracy and size than coarse-grained sparse CNNs. Due to irregular data structures and unbalanced computation loads, fine-grained sparse CNNs struggle to fully leverage the performance advantages of computation and storage on general-purpose edge hardware. However, existing custom sparse accelerators are designed from the perspective of emulating a balanced load by software or computational strategies, neglecting the exploration of the computing architecture’s adaptability and parallelism for fine-grained sparse models. To address these challenges, a cross-mesh NoC-based accelerator architecture is proposed. This architecture aligns with the irregular characteristics of fine-grained sparse CNN weights and enhances the spatio-temporal parallelism of fine-grained sparse CNNs. First, a sparse multiplier unit (SMU) array and an adder array are designed to enable parallel execution of convolution multiplication and accumulation operations. Then, element-wise unroll-based nonzero weight multiplication is mapped to the SMU array to provide more flexible spatial parallelism. A horizontal and vertical cross-mesh NoC is proposed for flexible dataflow scheduling between the SMU and adder arrays to further improve temporal parallelism. This architecture allows the multiplication and accumulation operations in convolution to be decoupled and pipelined with negligible latency. Finally, the proposed accelerator architecture is implemented on the ZU9EG platform. The experimental results show that the proposed accelerator achieves frame rates of 509.9, 249.3, 100.7, 48.4, and 168.9 frames per second (FPS) for AlexNet, VGG-16, ResNet-18, MobileNet-v2, and EfficientNet, respectively. Compared with related works, this accelerator achieves inference speed and energy efficiency improvements of 1.1 $\times$ $\sim$ 36.1 $\times$ and 2.4 $\times$ $\sim$ 13.4 $\times$ , respectively.

Exploiting Sparsity to Accelerate Fully Connected Layers of CNN-Based Applications on Mobile SoCs

Deep Neural Network Acceleration with Sparse Prediction Layers

Single-shot Pruning and Quantization for Hardware-Friendly Neural Network Acceleration

A GPU-based high-performance optimization method of sparse convolutional neural networks

Sensitivity-Oriented Layer-Wise Acceleration and Compression for Convolutional Neural Network.

Performance Aware Convolutional Neural Network Channel Pruning for Embedded GPUs

SparseTrain: Exploiting Dataflow Sparsity for Efficient Convolutional Neural Networks Training

Eyelet: A Cross-Mesh NoC-Based Fine-Grained Sparse CNN Accelerator for Spatio-Temporal Parallel Computing Optimization

Accelerating convolutional neural network by exploiting sparsity on GPUs

A Sparse CNN Accelerator for Eliminating Redundant Computations in Intra- and Inter-Convolutional/Pooling Layers

Accelerating CNN on mobile GPU

Accelerating Sparse CNN Inference on GPUs with Performance-Aware Weight Pruning

A Mixed-Pruning Based Framework for Embedded Convolutional Neural Network Acceleration.

Efficient Inference for Pruned CNN Models on Mobile Devices With Holistic Sparsity Alignment

An Efficient Accelerator for Multiple Convolutions From the Sparsity Perspective

SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks

Convolution Without Multiplication: A General Speed Up Strategy for CNNs

Enabling High Performance Deep Learning Networks on Embedded Systems

SparseByteNN: A Novel Mobile Inference Acceleration Framework Based on Fine-Grained Group Sparsity

A Parallel Processing CNN Accelerator on Embedded Devices Based on Optimized MobileNet

On pruned, quantized and compact CNN architectures for vision applications: an empirical study