Abstract:The tensor cores in modern GPUs lead to significant performance improvement in matrix multiplication, which is the primary operation in deep learning. However, existing hardware architectures face unstructured sparsity in deep learning, resulting in algorithm inflexibility and hardware inefficiency. The previous tensor core architecture requires matrices to be pruned into 2:4 sparse patterns, leading to algorithm inflexibility. Customized accelerators introduce extra architectures (e.g., interconnection networks for dynamic data routing or buffers for avoiding data conflicts) for unstructured sparse matrices, leading to hardware inefficiency. To tackle the contradiction between algorithm inflexibility and hardware inefficiency, we propose Two-level Sparsity Tensor Core (TSTC) in this paper. TSTC points out that the unstructured sparsity which enables algorithm flexibility can be maintained at the coarse-grained level, while hardware efficiency which requires structured sparsity can be ensured at the fine-grained level. For algorithm flexibility, we propose Flexible Sparse Block (FSB) pattern. FSB enables unstructured sparse matrices can be divided into fine-grained blocks with different structured sparsity. As a result, using FSB leads to up to 7.29x speed up compared with other formats. For hardware efficiency, we propose Dynamic Extendible Reduction Network (DERN). DERN enables different structured sparse reductions by only extending the data width on the standard reduction network without introducing interconnections or buffers. DERN enables TSTC to achieve 7.19x more energy savings under a similar speed. We also propose the whole flow, which can automatically deploy different sparse deep learning algorithms to TSTC. According to extensive experiments, TSTC achieves 1.24 x ~7.69 x speedup and 3.68 x~4.17 x energy savings than the tensor core and the SOTA customized accelerator.

GTCO: Graph and Tensor Co-Design for Transformer-Based Image Recognition on Tensor Cores

AutoGTCO: Graph and Tensor Co-Optimize for Image Recognition with Transformers on GPU

Heterogeneous Programming and Optimization of Gyrokinetic Toroidal Code and Large-Scale Performance Test on TH-1A.

TC-GNN: Bridging Sparse GNN Computation and Dense Tensor Cores on GPUs

TSCompiler: Efficient Compilation Framework for Dynamic-Shape Models

An Optimization Toolchain Design Of Deep Learning Deployment Based On Heterogeneous Computing Platform

TSTC: Two-Level Sparsity Tensor Core Enabling Both Algorithm Flexibility and Hardware Efficiency

GCoD: Graph Convolutional Network Acceleration via Dedicated Algorithm and Accelerator Co-Design

Graph neural networks with configuration cross-attention for tensor compilers

High-Performance Tensor Learning Primitives Using GPU Tensor Cores

Scaling Deep Learning Computation over the Inter-Core Connected Intelligence Processor with T10

BCB-SpTC: An Efficient Sparse High-Dimensional Tensor Contraction Employing Tensor Core Acceleration

PowerFusion: A Tensor Compiler with Explicit Data Movement Description and Instruction-level Graph IR

A Novel Parallel Algorithm for Sparse Tensor Matrix Chain Multiplication via TCU-Acceleration

A Scalable and Effective Alternative to Graph Transformers

High-Performance Tensor-Train Primitives Using GPU Tensor Cores

Exploring Approximation and Dataflow Co-Optimization for Scalable Transformer Inference Architecture on the Edge

PCGCN: Partition-Centric Processing for Accelerating Graph Convolutional Network

Scalable CP Decomposition for Tensor Learning using GPU Tensor Cores

G10: Enabling An Efficient Unified GPU Memory and Storage Architecture with Smart Tensor Migrations

Full Stack Optimization of Transformer Inference: a Survey