Abstract:The performance of sparse iterative solvers is typically limited by sparse matrix-vector multiplication, which is itself limited by memory system and network performance. As the gap between computation and communication speed continues to widen, these traditional sparse methods will suffer. In this paper we focus on an alternative building block for sparse iterative solvers, the “matrix powers ker-nel” [x, Ax, $\rm A^{2}x$, …, $\rm A^{k}x$], and show that by organizing computations around this kernel, we can achieve near-minimal communication costs. We consider communication very broadly as both network communication in parallel code and memory hierarchy access in sequential code. In particular, we introduce a parallel algorithm for which the number of messages (total latency cost) is independent of the power k, and a sequential algorithm, that reduces both the number and volume of accesses, so that it is independent of k in both latency and bandwidth costs. This is part of a larger project to develop “communication-avoiding Krylov subspace methods,” which also addresses the numerical issues associated with these methods. Our algorithms work for general sparse matrices that “partition well”. We introduce parallel performance models of matrices arising from 2D and 3D problems and show predicted speedups over a conventional algorithm of up to $7x$ on a Petaflop-scale machine and up to $22x$ on computation across the Grid. Analogous sequential performance models of the same problems predict speedups over a conventional algorithm of up to $10x$ on an out-of-core implemen-tation, and up to $2.5x$ when we use our ideas to reduce off-chip latency and bandwidth to DRAM. Finally, we validate the model on an out-of-core sequential implementation and measured a speedup of over $3x$, which is close to the predicted speedup.

A Communication Avoiding and Reducing Algorithm for Symmetric Eigenproblem for Very Small Matrices

A Communication-Avoiding Parallel Algorithm for the Symmetric Eigenvalue Problem

Avoiding communication in sparse matrix computations

Improving the Scalability of A Symmetric Iterative Eigensolver for Multi-Core Platforms

Collective Communication Optimization for Solving Linear Algebraic Equations

Reducing the Symmetric Matrix Eigenvalue Problem to Matrix Multiplications

Communication-Optimal Parallel Algorithm for Strassen's Matrix Multiplication

A Method for Solving Large Sparse Unsymmetric Linear Equations

A Parallel Algorithm Based on Galerkin Theory for Block-Tridiagonal Linear Systems

Tridiagonalization of a dense symmetric matrix on multiple GPUs and its application to symmetric eigenvalue problems

A Parallel Iterative Method for Solving Periodical Block-Tridiagonal Linear Equations.

A Memory Reduction Scheme for Parallel Finite Element Method Sparse Matrix Equation Solving

Implementing the Jacobi Algorithm for Solving Eigenvalues of Symmetric Matrices with CUDA

Performance Optimization for Sparse A(T)Ax in Parallel on Multicore Cpu

Orthogonal layers of parallelism in large-scale eigenvalue computations

A Shift Selection Strategy for Parallel Shift-Invert Spectrum Slicing in Symmetric Self-Consistent Eigenvalue Computation

Scalable Hierarchical Parallel Algorithm for the Solution of Super Large-Scale Sparse Linear Equations

Highly Efficient Parallel Direct Solver for Solving Dense Complex Matrix Equations from Method of Moments

Communication Lower Bounds and Optimal Algorithms for Symmetric Matrix Computations

On the Parallel I/O Optimality of Linear Algebra Kernels: Near-Optimal Matrix Factorizations

A High Performance Block Eigensolver for Nuclear Configuration Interaction Calculations.