Abstract:We present novel algorithmic solutions together with implementation details utilizing non-Abelian symmetries in order to boost the current limits of tensor network state algorithms on high performance computing infrastructure. In our in-house developed hybrid CPU-multiGPU solution scheduling is decentralized, threads are autonomous and inter-thread communications are solely limited to interactions with globally visible lock-free constructs. Our custom tailored virtual memory management ensures data is produced with high spatial locality, which together with the use of specific sequences of strided batched matrix operations translates to significantly higher overall throughput. In order to lower IO overhead, an adaptive buffering technique is used to dynamically match the level of data abstraction, at which cache repositories are built and reused, to system resources. The non-Abelian symmetry related tensor algebra based on Wigner-Eckhart theorem is fully detached from the conventional tensor network layer, thus massively parallel matrix and tensor operations can be performed without additional overheads. Altogether, we have achieved an order of magnitude increase in performance with respect to results reported in <a class="link-https" data-arxiv-id="2305.05581" href="https://arxiv.org/abs/2305.05581">arXiv:2305.05581</a> in terms of computational complexity and at the same time a factor of three to six in the actual performance measured in TFLOPS. Benchmark results are presented on Hilbert space dimensions up to $2.88\times10^{36}$ obtained via large-scale SU(2) spin adapted density matrix renormalization group simulations on selected strongly correlated molecular systems. These demonstrate the utilization of NVIDIA's highly specialized tensor cores, leading to performance around 110 TFLOPS on a single node supplied with eight NVIDIA A100 devices. In comparison to U(1) implementations with matching accuracy, our solution has an estimated effective performance of 250-500 TFLOPS.

T2S-Tensor: Productively Generating High-Performance Spatial Hardware for Dense Tensor Computations

Programmatic Control of a Compiler for Generating High-performance Spatial Hardware

Heterogeneous Programming and Optimization of Gyrokinetic Toroidal Code and Large-Scale Performance Test on TH-1A.

Software for Sparse Tensor Decomposition on Emerging Computing Architectures

Automatic Generation of Spatial Accelerator for Tensor Algebra

TensorLib - A Spatial Accelerator Generation Framework for Tensor Algebra.

The tensor algebra compiler

Sgap: Towards Efficient Sparse Tensor Algebra Compilation for GPU

TSTC: Two-Level Sparsity Tensor Core Enabling Both Algorithm Flexibility and Hardware Efficiency

Systolic Computing on GPUs for Productive Performance

A Novel Parallel Algorithm for Sparse Tensor Matrix Chain Multiplication via TCU-Acceleration

High-Performance Generalized Tensor Operations

Enabling Tensor Language Model to Assist in Generating High-Performance Tensor Programs for Deep Learning.

High Performance Unstructured SpMM Computation Using Tensor Cores

AMOS: enabling <u>a</u>utomatic <u>m</u>apping for tensor computations <u>o</u>n <u>s</u>patial accelerators with hardware abstraction

Performance Optimization for Sparse A(T)Ax in Parallel on Multicore Cpu

Spatial: a language and compiler for application accelerators

BCB-SpTC: An Efficient Sparse High-Dimensional Tensor Contraction Employing Tensor Core Acceleration

Analyzing the Performance Portability of Tensor Decomposition

Efficient Utilization of Multi-Threading Parallelism on Heterogeneous Systems for Sparse Tensor Contraction

Boosting the effective performance of massively parallel tensor network state algorithms on hybrid CPU-GPU based architectures via non-Abelian symmetries