Abstract:As the model scale of Deep Neural Networks (DNNs) increases, the memory and computational cost of DNNs become overwhelmingly large. Sparse Deep Neural Networks (SpDNNs) are promising to cope with this challenge by using fewer weights while preserving the accuracy. However, the sparsity nature of SpDNN models makes it difficult to run efficiently on GPUs. To stimulate technical advances for improving the efficiency of SpDNN inference, the MIT/IEEE/Amazon GraphChallenge proposes the SpDNN Challenge in 2019. In this paper, we present a hybrid compute model to improve the efficiency of Sparse Matrix Multiplications (SpMMs), the core computation of SpDNN inference. First, the given sparse weight matrix will be divided to generate many (sparse and dense) submatrices. For sparse submatrices, we leverage compile-time data embedding to compile the sparse data together with their corresponding computations into instructions and hence the number of random accesses can be reduced significantly. For dense submatrices, we follow the traditional computing mode where the data is obtained from the memory to exploit the high memory bandwidth of GPU. This hybrid compute model effectively balances the memory and instruction bottlenecks, and offers more scheduling opportunities to overlap computing operations and memory accesses on GPU. To determine whether a sub matrix is sparse, we present a cost model to estimate its time cost under the traditional computing mode and the data-embedded computing mode in an accurate and efficient manner. Once the computing mode for all submatrices is determined, customized codes will be generated for the SpDNN inference. Experimental results on the SpDNN Challenge benchmarks show that our approach achieves up to 197.86 tera-edges per second inference throughput on a single NVIDIA A100 GPU. Compared to the 2021 and 2020 champions, our approach offers up to 6.37x and 89.94x speedups on a single GPU, respectively. We also implement a 16-GPU version, showing up to 9.49x and 80.11x speedups over the former 16-GPU baselines of the 2021 and 2020 champions.

Dynamic Sparse Graph for Efficient Deep Learning.

Efficient Structure Slimming for Spiking Neural Networks

DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Vision Transformers

Neurogenesis Dynamics-inspired Spiking Neural Network Training Acceleration

SparseNN: an Energy-Efficient Neural Network Accelerator Exploiting Input and Output Sparsity

Performance of Training Sparse Deep Neural Networks on GPUs

Dynamic Spiking Framework for Graph Neural Networks

DSTC: Dual-Side Sparsity Tensor Core for DNNs Acceleration on Modern GPU Architectures

Compressing Deep Neural Networks With Sparse Matrix Factorization

Enabling Massive Deep Neural Networks with the GraphBLAS

DynSNN: A Dynamic Approach to Reduce Redundancy in Spiking Neural Networks

Towards Fast GPU-based Sparse DNN Inference: A Hybrid Compute Model

Sparse Decomposition of Graph Neural Networks

Flexible and Scalable Deep Dendritic Spiking Neural Networks with Multiple Nonlinear Branching

Efficient N:M Sparse DNN Training Using Algorithm, Architecture, and Dataflow Co-Design

Deep Adaptive Network: An Efficient Deep Neural Network with Sparse Binary Connections

Smart-DNN: Efficiently Reducing the Memory Requirements of Running Deep Neural Networks on Resource-constrained Platforms

Accelerating Deep Neural Networks via Semi-Structured Activation Sparsity

Balanced Sparsity for Efficient DNN Inference on GPU

Dynamic Spiking Graph Neural Networks.

Efficient Deep Learning Inference Based on Model Compression.