Abstract:The tensor cores in modern GPUs lead to significant performance improvement in matrix multiplication, which is the primary operation in deep learning. However, existing hardware architectures face unstructured sparsity in deep learning, resulting in algorithm inflexibility and hardware inefficiency. The previous tensor core architecture requires matrices to be pruned into 2:4 sparse patterns, leading to algorithm inflexibility. Customized accelerators introduce extra architectures (e.g., interconnection networks for dynamic data routing or buffers for avoiding data conflicts) for unstructured sparse matrices, leading to hardware inefficiency. To tackle the contradiction between algorithm inflexibility and hardware inefficiency, we propose Two-level Sparsity Tensor Core (TSTC) in this paper. TSTC points out that the unstructured sparsity which enables algorithm flexibility can be maintained at the coarse-grained level, while hardware efficiency which requires structured sparsity can be ensured at the fine-grained level. For algorithm flexibility, we propose Flexible Sparse Block (FSB) pattern. FSB enables unstructured sparse matrices can be divided into fine-grained blocks with different structured sparsity. As a result, using FSB leads to up to 7.29x speed up compared with other formats. For hardware efficiency, we propose Dynamic Extendible Reduction Network (DERN). DERN enables different structured sparse reductions by only extending the data width on the standard reduction network without introducing interconnections or buffers. DERN enables TSTC to achieve 7.19x more energy savings under a similar speed. We also propose the whole flow, which can automatically deploy different sparse deep learning algorithms to TSTC. According to extensive experiments, TSTC achieves 1.24 x ~7.69 x speedup and 3.68 x~4.17 x energy savings than the tensor core and the SOTA customized accelerator.

Cardinality Sparsity: Applications in Matrix-Matrix Multiplications and Machine Learning

Two Sparsities Are Better Than One: Unlocking the Performance Benefits of Sparse-Sparse Networks

The Time Complexity of Fully Sparse Matrix Multiplication

Kronecker-Basis-Representation Based Tensor Sparsity and Its Applications to Tensor Recovery.

An Overview of Computational Sparse Models and Their Applications in Artificial Intelligence

Masked Matrix Multiplication for Emergent Sparsity

Sparse Coding and Dictionary Learning for Symmetric Positive Definite Matrices: A Kernel Approach

Arrow Matrix Decomposition: A Novel Approach for Communication-Efficient Sparse Matrix Multiplication

Tensor Core-Adapted Sparse Matrix Multiplication for Accelerating Sparse Deep Neural Networks

Fast inference with Kronecker-sparse matrices

Coded Sparse Matrix Multiplication

TSTC: Two-Level Sparsity Tensor Core Enabling Both Algorithm Flexibility and Hardware Efficiency

Tensor-Based Sparsity Order Estimation for Big Data Applications

MMSparse: 2D Partitioning of Sparse Matrix Based on Mathematical Morphology

Tensor-Sparsity of Solutions to High-Dimensional Elliptic Partial Differential Equations

Less is More -- Towards parsimonious multi-task models using structured sparsity

Diagonally-Addressed Matrix Nicknack: How to improve SpMV performance

Exploring the Regularity of Sparse Structure in Convolutional Neural Networks

Increasing the Efficiency of Sparse Matrix-Matrix Multiplication with a 2.5D Algorithm and One-Sided MPI

A Novel Tensor Ring Sparsity Measurement for Image Completion

DSTC: Dual-Side Sparsity Tensor Core for DNNs Acceleration on Modern GPU Architectures