Abstract:As the model scale of Deep Neural Networks (DNNs) increases, the memory and computational cost of DNNs become overwhelmingly large. Sparse Deep Neural Networks (SpDNNs) are promising to cope with this challenge by using fewer weights while preserving the accuracy. However, the sparsity nature of SpDNN models makes it difficult to run efficiently on GPUs. To stimulate technical advances for improving the efficiency of SpDNN inference, the MIT/IEEE/Amazon GraphChallenge proposes the SpDNN Challenge in 2019. In this paper, we present a hybrid compute model to improve the efficiency of Sparse Matrix Multiplications (SpMMs), the core computation of SpDNN inference. First, the given sparse weight matrix will be divided to generate many (sparse and dense) submatrices. For sparse submatrices, we leverage compile-time data embedding to compile the sparse data together with their corresponding computations into instructions and hence the number of random accesses can be reduced significantly. For dense submatrices, we follow the traditional computing mode where the data is obtained from the memory to exploit the high memory bandwidth of GPU. This hybrid compute model effectively balances the memory and instruction bottlenecks, and offers more scheduling opportunities to overlap computing operations and memory accesses on GPU. To determine whether a sub matrix is sparse, we present a cost model to estimate its time cost under the traditional computing mode and the data-embedded computing mode in an accurate and efficient manner. Once the computing mode for all submatrices is determined, customized codes will be generated for the SpDNN inference. Experimental results on the SpDNN Challenge benchmarks show that our approach achieves up to 197.86 tera-edges per second inference throughput on a single NVIDIA A100 GPU. Compared to the 2021 and 2020 champions, our approach offers up to 6.37x and 89.94x speedups on a single GPU, respectively. We also implement a 16-GPU version, showing up to 9.49x and 80.11x speedups over the former 16-GPU baselines of the 2021 and 2020 champions.

DSTC: Dual-Side Sparsity Tensor Core for DNNs Acceleration on Modern GPU Architectures

Dual-side Sparse Tensor Core

TSTC: Two-Level Sparsity Tensor Core Enabling Both Algorithm Flexibility and Hardware Efficiency

RM-STC: Row-Merge Dataflow Inspired GPU Sparse Tensor Core for Energy-Efficient Sparse Acceleration.

Accelerating Sparse Deep Neural Network Inference Using GPU Tensor Cores

Towards Fast GPU-based Sparse DNN Inference: A Hybrid Compute Model

THETA: A High-Efficiency Training Accelerator for DNNs with Triple-Side Sparsity Exploration

A fine-grained mixed precision DNN accelerator using a two-stage big-little core RISC-V MCU.

Balanced Sparsity for Efficient DNN Inference on GPU

Accelerating Sparse DNN Models Without Hardware-Support Via Tile-Wise Sparsity

SparseNN: an Energy-Efficient Neural Network Accelerator Exploiting Input and Output Sparsity

Dynamic Sparse Graph for Efficient Deep Learning.

Sparse Systolic Tensor Array for Efficient CNN Hardware Acceleration

An Efficient Hardware Architecture for DNN Training by Exploiting Triple Sparsity

Releasing the Potential of Tensor Core for Unstructured SpMM Using Tiled-CSR Format

Exploiting Dynamic Bit Sparsity in Activation for Deep Neural Network Acceleration.

Abstracting Sparse DNN Acceleration via Structured Sparse Tensor Decomposition

Taming Unstructured Sparsity on GPUs Via Latency-Aware Optimization

A Fine-Grained Sparse Accelerator for Multi-Precision DNN.

TSTC: Enabling Efficient Training Via Structured Sparse Tensor Compilation

Shfl-BW: Accelerating Deep Neural Network Inference with Tensor-Core Aware Weight Pruning