Abstract:Communication-avoiding algorithms for Linear Algebra have become increasingly popular, in particular for distributed memory architectures. In practice, these algorithms assume that the data is already distributed in a specific way, thus making data reshuffling a key to use them. For performance reasons, a straightforward all-to-all exchange must be avoided. Here, we show that process relabeling (i.e. permuting processes in the final layout) can be used to obtain communication optimality for data reshuffling, and that it can be efficiently found by solving a Linear Assignment Problem (Maximum Weight Bipartite Perfect Matching). Based on this, we have developed a Communication-Optimal Shuffle and Transpose Algorithm (COSTA): this highly-optimised algorithm implements $A=\alpha\cdot \operatorname{op}(B) + \beta \cdot A,\ \operatorname{op} \in \{\operatorname{transpose}, \operatorname{conjugate-transpose}, \operatorname{identity}\}$ on distributed systems, where $A, B$ are matrices with potentially different (distributed) layouts and $\alpha, \beta$ are scalars. COSTA can take advantage of the communication-optimal process relabeling even for heterogeneous network topologies, where latency and bandwidth differ among nodes. The implementation not only outperforms the best available ScaLAPACK redistribute and transpose routines multiple times, but is also able to deal with more general matrix layouts, in particular it is not limited to block-cyclic layouts. Finally, we use COSTA to integrate a communication-optimal matrix multiplication algorithm into the CP2K quantum chemistry simulation package. This way, we show that COSTA can be used to unlock the full potential of recent Linear Algebra algorithms in applications by facilitating interoperability between algorithms with a wide range of data layouts, in addition to bringing significant redistribution speedups.

Communication-Optimal Parallel Algorithm for Strassen's Matrix Multiplication

Avoiding communication in sparse matrix computations

Communication Lower Bounds and Optimal Algorithms for Symmetric Matrix Computations

On the Parallel I/O Optimality of Linear Algebra Kernels: Near-Optimal Matrix Factorizations

On Parallelizing Matrix Multiplication by the Column-Row Method

Graph Expansion Analysis for Communication Costs of Fast Rectangular Matrix Multiplication

A Communication-Avoiding Parallel Algorithm for the Symmetric Eigenvalue Problem

Increasing the Efficiency of Sparse Matrix-Matrix Multiplication with a 2.5D Algorithm and One-Sided MPI

Communication Lower Bounds and Optimal Algorithms for Multiple Tensor-Times-Matrix Computation

Improving the Space-Time Efficiency of Matrix Multiplication Algorithms

A Work-Efficient Parallel Sparse Matrix-Sparse Vector Multiplication Algorithm

Implementing Communication-Optimal Parallel and Sequential QR Factorizations

COSTA: Communication-Optimal Shuffle and Transpose Algorithm with Process Relabeling

Fast and Processor Efficient Parallel Matrix Multiplication Algorithms on a Linear Array with a Reconfigurable Pipelined Bus System

Improving the Space-Time Efficiency of Processor-Oblivious Matrix Multiplication Algorithms

A sparsity-aware distributed-memory algorithm for sparse-sparse matrix multiplication

A Communication Avoiding and Reducing Algorithm for Symmetric Eigenproblem for Very Small Matrices

Implementing Strassen's Algorithm with BLIS

Fast and Practical Strassen's Matrix Multiplication using FPGAs

Arrow Matrix Decomposition: A Novel Approach for Communication-Efficient Sparse Matrix Multiplication

Collective Communication Optimization for Solving Linear Algebraic Equations