Abstract:Communication-avoiding algorithms for Linear Algebra have become increasingly popular, in particular for distributed memory architectures. In practice, these algorithms assume that the data is already distributed in a specific way, thus making data reshuffling a key to use them. For performance reasons, a straightforward all-to-all exchange must be avoided. Here, we show that process relabeling (i.e. permuting processes in the final layout) can be used to obtain communication optimality for data reshuffling, and that it can be efficiently found by solving a Linear Assignment Problem (Maximum Weight Bipartite Perfect Matching). Based on this, we have developed a Communication-Optimal Shuffle and Transpose Algorithm (COSTA): this highly-optimised algorithm implements $A=\alpha\cdot \operatorname{op}(B) + \beta \cdot A,\ \operatorname{op} \in \{\operatorname{transpose}, \operatorname{conjugate-transpose}, \operatorname{identity}\}$ on distributed systems, where $A, B$ are matrices with potentially different (distributed) layouts and $\alpha, \beta$ are scalars. COSTA can take advantage of the communication-optimal process relabeling even for heterogeneous network topologies, where latency and bandwidth differ among nodes. The implementation not only outperforms the best available ScaLAPACK redistribute and transpose routines multiple times, but is also able to deal with more general matrix layouts, in particular it is not limited to block-cyclic layouts. Finally, we use COSTA to integrate a communication-optimal matrix multiplication algorithm into the CP2K quantum chemistry simulation package. This way, we show that COSTA can be used to unlock the full potential of recent Linear Algebra algorithms in applications by facilitating interoperability between algorithms with a wide range of data layouts, in addition to bringing significant redistribution speedups.

Communication Lower Bounds and Optimal Algorithms for Multiple Tensor-Times-Matrix Computation

Communication Lower Bounds and Optimal Algorithms for Symmetric Matrix Computations

Communication-Optimal Parallel Algorithm for Strassen's Matrix Multiplication

A Novel Parallel Algorithm for Sparse Tensor Matrix Chain Multiplication via TCU-Acceleration

Communication Avoiding Block Low-Rank Parallel Multifrontal Triangular Solve with Many Right-Hand Sides

Tensor Decompositions and Adiabatic Quantum Computing for Discovering Practical Matrix Multiplication Algorithms

Statistical-Computational Trade-offs in Tensor PCA and Related Problems via Communication Complexity

Collective Communication Optimization for Solving Linear Algebraic Equations

Optimizing Distributed Tensor Contractions using Node-Aware Processor Grids

Graph Expansion Analysis for Communication Costs of Fast Rectangular Matrix Multiplication

Avoiding communication in sparse matrix computations

On the Parallel I/O Optimality of Linear Algebra Kernels: Near-Optimal LU Factorization

Parallel Randomized Tucker Decomposition Algorithms

Improving the Space-Time Efficiency of Matrix Multiplication Algorithms

Deinsum: Practically I/O Optimal Multilinear Algebra

On the Parallel I/O Optimality of Linear Algebra Kernels: Near-Optimal Matrix Factorizations

T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives

Efficient algorithms for Tucker decomposition via approximate matrix multiplication

Reducing Computational Complexity of Tensor Contractions via Tensor-Train Networks

COSTA: Communication-Optimal Shuffle and Transpose Algorithm with Process Relabeling

Improving the Space-Time Efficiency of Processor-Oblivious Matrix Multiplication Algorithms