Abstract:We present novel algorithmic solutions together with implementation details utilizing non-Abelian symmetries in order to boost the current limits of tensor network state algorithms on high performance computing infrastructure. In our in-house developed hybrid CPU-multiGPU solution scheduling is decentralized, threads are autonomous and inter-thread communications are solely limited to interactions with globally visible lock-free constructs. Our custom tailored virtual memory management ensures data is produced with high spatial locality, which together with the use of specific sequences of strided batched matrix operations translates to significantly higher overall throughput. In order to lower IO overhead, an adaptive buffering technique is used to dynamically match the level of data abstraction, at which cache repositories are built and reused, to system resources. The non-Abelian symmetry related tensor algebra based on Wigner-Eckhart theorem is fully detached from the conventional tensor network layer, thus massively parallel matrix and tensor operations can be performed without additional overheads. Altogether, we have achieved an order of magnitude increase in performance with respect to results reported in <a class="link-https" data-arxiv-id="2305.05581" href="https://arxiv.org/abs/2305.05581">arXiv:2305.05581</a> in terms of computational complexity and at the same time a factor of three to six in the actual performance measured in TFLOPS. Benchmark results are presented on Hilbert space dimensions up to $2.88\times10^{36}$ obtained via large-scale SU(2) spin adapted density matrix renormalization group simulations on selected strongly correlated molecular systems. These demonstrate the utilization of NVIDIA's highly specialized tensor cores, leading to performance around 110 TFLOPS on a single node supplied with eight NVIDIA A100 devices. In comparison to U(1) implementations with matching accuracy, our solution has an estimated effective performance of 250-500 TFLOPS.

A Parallel Sparse Tensor Benchmark Suite on CPUs and GPUs

PASTA: A Parallel Sparse Tensor Algorithm Benchmark Suite

A High-Performance Sparse Tensor Algebra Compiler in Multi-Level IR

Benchmarking GPU Tensor Cores on General Matrix Multiplication Kernels through CUTLASS

Software for Sparse Tensor Decomposition on Emerging Computing Architectures

Efficient and effective sparse tensor reordering

Performance Optimization for Sparse A(T)Ax in Parallel on Multicore Cpu

A Novel Parallel Algorithm for Sparse Tensor Matrix Chain Multiplication via TCU-Acceleration

On Parallel Solution of Sparse Triangular Linear Systems in CUDA

Optimizing Sparse Tensor Times Matrix on Multi-Core and Many-Core Architectures

Optimizing Sparse Tensor Times Matrix on GPUs

Parallel Sparse Tensor Decomposition in Chapel

Galley: Modern Query Optimization for Sparse Tensor Programs

Polyhedral Specification and Code Generation of Sparse Tensor Contraction with Co-iteration

T2S-Tensor: Productively Generating High-Performance Spatial Hardware for Dense Tensor Computations

Optimizing Algorithm of Sparse Linear Systems on GPU

Efficient Sparse-Dense Matrix-Matrix Multiplication on GPUs Using the Customized Sparse Storage Format

Shared Memory Parallelization of MTTKRP for Dense Tensors

Analyzing the Performance Portability of Tensor Decomposition

Benchmarking State-of-the-Art Deep Learning Software Tools

Boosting the effective performance of massively parallel tensor network state algorithms on hybrid CPU-GPU based architectures via non-Abelian symmetries