Abstract:The tensor cores in modern GPUs lead to significant performance improvement in matrix multiplication, which is the primary operation in deep learning. However, existing hardware architectures face unstructured sparsity in deep learning, resulting in algorithm inflexibility and hardware inefficiency. The previous tensor core architecture requires matrices to be pruned into 2:4 sparse patterns, leading to algorithm inflexibility. Customized accelerators introduce extra architectures (e.g., interconnection networks for dynamic data routing or buffers for avoiding data conflicts) for unstructured sparse matrices, leading to hardware inefficiency. To tackle the contradiction between algorithm inflexibility and hardware inefficiency, we propose Two-level Sparsity Tensor Core (TSTC) in this paper. TSTC points out that the unstructured sparsity which enables algorithm flexibility can be maintained at the coarse-grained level, while hardware efficiency which requires structured sparsity can be ensured at the fine-grained level. For algorithm flexibility, we propose Flexible Sparse Block (FSB) pattern. FSB enables unstructured sparse matrices can be divided into fine-grained blocks with different structured sparsity. As a result, using FSB leads to up to 7.29x speed up compared with other formats. For hardware efficiency, we propose Dynamic Extendible Reduction Network (DERN). DERN enables different structured sparse reductions by only extending the data width on the standard reduction network without introducing interconnections or buffers. DERN enables TSTC to achieve 7.19x more energy savings under a similar speed. We also propose the whole flow, which can automatically deploy different sparse deep learning algorithms to TSTC. According to extensive experiments, TSTC achieves 1.24 x ~7.69 x speedup and 3.68 x~4.17 x energy savings than the tensor core and the SOTA customized accelerator.

SpTFS: Sparse Tensor Format Selection for MTTKRP Via Deep Learning

Input-aware Sparse Tensor Storage Format Selection for Optimizing MTTKRP

Directional Tensor Product Complex Tight Framelets for Compressed Sensing MRI Reconstruction

TSTC: Two-Level Sparsity Tensor Core Enabling Both Algorithm Flexibility and Hardware Efficiency

A Novel Parallel Algorithm for Sparse Tensor Matrix Chain Multiplication via TCU-Acceleration

Sparse MTTKRP Acceleration for Tensor Decomposition on GPU

Software for Sparse Tensor Decomposition on Emerging Computing Architectures

Shared Memory Parallelization of MTTKRP for Dense Tensors

POSTER: Optimizing Sparse Tensor Contraction with Revisiting Hash Table Design.

Distributed memory parallel adaptive tensor-train cross approximation

Efficient Utilization of Multi-Threading Parallelism on Heterogeneous Systems for Sparse Tensor Contraction

BCB-SpTC: An Efficient Sparse High-Dimensional Tensor Contraction Employing Tensor Core Acceleration

Tensor Core-Adapted Sparse Matrix Multiplication for Accelerating Sparse Deep Neural Networks

VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores

TEC: Tensor Ensemble Classifier for Big Data

Sparse Tensor PCA via Tensor Decomposition for Unsupervised Feature Selection

Scalable Tucker Factorization for Sparse Tensors - Algorithms and Discoveries

Sparse Tucker Tensor Decomposition on a Hybrid FPGA-CPU Platform

Latent Matrices for Tensor Network Decomposition and to Tensor Completion

Efficient and effective sparse tensor reordering

T2S-Tensor: Productively Generating High-Performance Spatial Hardware for Dense Tensor Computations