Abstract:Sparsity is becoming arguably the most critical dimension to explore for efficiency and scalability as deep learning models grow significantly larger. Particularly, pruning is a common method to reduce redundant computations in attention-based and convolution-based models. The induced sparse matrix multiplication (SpMM) normally requires domain-specific hardware architecture (DSA) to eliminate unnecessary zero-valued computations. However, generating an optimal kernel code for SpMM on general-purpose and ISA-based spatial accelerators without changing the hardware architecture is still an open problem.In this paper, we propose a compiler plug-in named SpMMPlu, which can extend the representation and optimization ability for SpMM in current deep learning compiler frameworks that only support dense matrix multiplication. The key of SpMMPlu is a flexible intermediate representation— Sparse IR, representing the SpMM with various sparsity patterns based on meta-ops with a multi-level structure. Meta-op takes abstraction of the hardware intrinsic as its minimum granularity, and the powerful optimizers of existing NN compiler backends (e.g., Auto-schedule in TVM, AKG in MindSpore) can be easily reused for its computational scheduling and code generation. Moreover, we propose a two-step (segmentation & grouping) method to achieve an efficient Sparse IR for each sparsity pattern. Only three passes are added in SpMMPlu to provide an automatic solution for SpMM kernel code generation. We embed SpMMPlu into MindSpore and do experiments on NVIDIA V100 GPU and Huawei Ascend 910 to verify its effectiveness and scalability. The results show that with SpMMPlu, MindSpore can support various sparsity patterns and deliver a 1.93× (on V100 GPU) and 2.21× (on AScend 910) speedup averagely compared to the dense counterpart.

High Performance Unstructured SpMM Computation Using Tensor Cores

Tensor Core-Adapted Sparse Matrix Multiplication for Accelerating Sparse Deep Neural Networks

A Novel Parallel Algorithm for Sparse Tensor Matrix Chain Multiplication via TCU-Acceleration

Design and Implementation of Adaptive SpMV Library for Multicore and Many-Core Architecture

Smat: An Input Adaptive Auto-Tuner For Sparse Matrix-Vector Multiplication

SMAT: An Input Adaptive Sparse Matrix-Vector Multiplication Auto-Tuner

Efficient sparse-matrix multi-vector product on GPUs

Accelerating Sparse Approximate Matrix Multiplication on GPUs

BCB-SpTC: An Efficient Sparse High-Dimensional Tensor Contraction Employing Tensor Core Acceleration

Parallel Efficient Sparse Matrix-Matrix Multiplication on Multicore Platforms

Efficient Sparse-Dense Matrix-Matrix Multiplication on GPUs Using the Customized Sparse Storage Format

Distributed-Memory Parallel Algorithms for Sparse Matrix and Sparse Tall-and-Skinny Matrix Multiplication

Efficient Utilization of Multi-Threading Parallelism on Heterogeneous Systems for Sparse Tensor Contraction

Dedicated Hardware Accelerators for Processing of Sparse Matrices and Vectors: A Survey

Heuristic Adaptability to Input Dynamics for SpMM on GPUs

EC-SpMM: Efficient Compilation of SpMM Kernel on GPUs.

Accelerating approximate matrix multiplication for near-sparse matrices on GPUs

SpMMPlu: A Compiler Plug-in with Sparse IR for Efficient Sparse Matrix Multiplication.

T2S-Tensor: Productively Generating High-Performance Spatial Hardware for Dense Tensor Computations

Scale-Free Sparse Matrix-Vector Multiplication on Many-Core Architectures

Performance Optimization for Sparse A(T)Ax in Parallel on Multicore Cpu