Abstract:Fine-grained sparse convolutional neural networks (CNNs) achieve a better trade-off between model accuracy and size than coarse-grained sparse CNNs. Due to irregular data structures and unbalanced computation loads, fine-grained sparse CNNs struggle to fully leverage the performance advantages of computation and storage on general-purpose edge hardware. However, existing custom sparse accelerators are designed from the perspective of emulating a balanced load by software or computational strategies, neglecting the exploration of the computing architecture’s adaptability and parallelism for fine-grained sparse models. To address these challenges, a cross-mesh NoC-based accelerator architecture is proposed. This architecture aligns with the irregular characteristics of fine-grained sparse CNN weights and enhances the spatio-temporal parallelism of fine-grained sparse CNNs. First, a sparse multiplier unit (SMU) array and an adder array are designed to enable parallel execution of convolution multiplication and accumulation operations. Then, element-wise unroll-based nonzero weight multiplication is mapped to the SMU array to provide more flexible spatial parallelism. A horizontal and vertical cross-mesh NoC is proposed for flexible dataflow scheduling between the SMU and adder arrays to further improve temporal parallelism. This architecture allows the multiplication and accumulation operations in convolution to be decoupled and pipelined with negligible latency. Finally, the proposed accelerator architecture is implemented on the ZU9EG platform. The experimental results show that the proposed accelerator achieves frame rates of 509.9, 249.3, 100.7, 48.4, and 168.9 frames per second (FPS) for AlexNet, VGG-16, ResNet-18, MobileNet-v2, and EfficientNet, respectively. Compared with related works, this accelerator achieves inference speed and energy efficiency improvements of 1.1 $\times$ $\sim$ 36.1 $\times$ and 2.4 $\times$ $\sim$ 13.4 $\times$ , respectively.

High Throughput CNN Inference and Training with In-Cache Computation

Neural Cache: Bit-Serial In-Cache Acceleration of Deep Neural Networks.

An Efficient Parallel CNN Inference Framework for Multi-zone Processor

An Efficient CNN Inference Accelerator Based on Intra- and Inter-Channel Feature Map Compression

Accelerating Convolutional Neural Network Inference Based on a Reconfigurable Sliced Systolic Array

Researching the CNN Collaborative Inference Mechanism for Heterogeneous Edge Devices

A High-performance Inference Accelerator Exploiting Patterned Sparsity in CNNs

A High Efficient Architecture for Convolution Neural Network Accelerator

Exploration of Balanced Design in Resource-Constrained Edge Device for Efficient CNNs

Accelerating Convolutional Neural Networks for Continuous Mobile Vision Via Cache Reuse.

IECA: An In-Execution Configuration CNN Accelerator With 30.55 GOPS/mm² Area Efficiency

Eyelet: A Cross-Mesh NoC-Based Fine-Grained Sparse CNN Accelerator for Spatio-Temporal Parallel Computing Optimization

An Enhanced Data Cache with In-Cache Processing Units for Convolutional Neural Network Accelerators

Many Hands Make Light Work: Accelerating Edge Inference via Multi-Client Collaborative Caching

LACS: A High-Computational-Efficiency Accelerator for CNNs

Memory optimization of CNN Heterogeneous Multi-Core Architecture

An Efficient Streaming Accelerator for Low Bit-Width Convolutional Neural Networks

A Unified Optimization Approach for CNN Model Inference on Integrated GPUs

A Reconfigurable Spatial Architecture for Energy-Efficient Inception Neural Networks

A Parallel Loading Based Accelerator for Convolution Neural Network

CAT: Compression-Aware Training for bandwidth reduction