Abstract:Communication-avoiding algorithms for Linear Algebra have become increasingly popular, in particular for distributed memory architectures. In practice, these algorithms assume that the data is already distributed in a specific way, thus making data reshuffling a key to use them. For performance reasons, a straightforward all-to-all exchange must be avoided. Here, we show that process relabeling (i.e. permuting processes in the final layout) can be used to obtain communication optimality for data reshuffling, and that it can be efficiently found by solving a Linear Assignment Problem (Maximum Weight Bipartite Perfect Matching). Based on this, we have developed a Communication-Optimal Shuffle and Transpose Algorithm (COSTA): this highly-optimised algorithm implements $A=\alpha\cdot \operatorname{op}(B) + \beta \cdot A,\ \operatorname{op} \in \{\operatorname{transpose}, \operatorname{conjugate-transpose}, \operatorname{identity}\}$ on distributed systems, where $A, B$ are matrices with potentially different (distributed) layouts and $\alpha, \beta$ are scalars. COSTA can take advantage of the communication-optimal process relabeling even for heterogeneous network topologies, where latency and bandwidth differ among nodes. The implementation not only outperforms the best available ScaLAPACK redistribute and transpose routines multiple times, but is also able to deal with more general matrix layouts, in particular it is not limited to block-cyclic layouts. Finally, we use COSTA to integrate a communication-optimal matrix multiplication algorithm into the CP2K quantum chemistry simulation package. This way, we show that COSTA can be used to unlock the full potential of recent Linear Algebra algorithms in applications by facilitating interoperability between algorithms with a wide range of data layouts, in addition to bringing significant redistribution speedups.

Efficient and effective sparse tensor reordering

Bringing Order to Sparsity: A Sparse Matrix Reordering Study on Multicore CPUs

A Parallel Sparse Tensor Benchmark Suite on CPUs and GPUs

Distributed-Memory Randomized Algorithms for Sparse Tensor CP Decomposition

Performance Optimization for Sparse A(T)Ax in Parallel on Multicore Cpu

Shared Memory Parallelization of MTTKRP for Dense Tensors

A Novel Parallel Algorithm for Sparse Tensor Matrix Chain Multiplication via TCU-Acceleration

Input-aware Sparse Tensor Storage Format Selection for Optimizing MTTKRP

Software for Sparse Tensor Decomposition on Emerging Computing Architectures

Automatic generation of efficient sparse tensor format conversion routines

POSTER: Optimizing Sparse Tensor Contraction with Revisiting Hash Table Design.

SpTFS: Sparse Tensor Format Selection for MTTKRP Via Deep Learning

Efficient $1$-bit tensor approximations

BALS: Blocked Alternating Least Squares for Parallel Sparse Matrix Factorization on GPUs

CANDECOMP/PARAFAC Decomposition of High-Order Tensors Through Tensor Reshaping

BOBA: A Parallel Lightweight Graph Reordering Algorithm with Heavyweight Implications

Polyhedral Specification and Code Generation of Sparse Tensor Contraction with Co-iteration

Fast Search of the Optimal Contraction Sequence in Tensor Networks

PASTA: A Parallel Sparse Tensor Algorithm Benchmark Suite

Efficient Utilization of Multi-Threading Parallelism on Heterogeneous Systems for Sparse Tensor Contraction

COSTA: Communication-Optimal Shuffle and Transpose Algorithm with Process Relabeling