Abstract:The performance of sparse iterative solvers is typically limited by sparse matrix-vector multiplication, which is itself limited by memory system and network performance. As the gap between computation and communication speed continues to widen, these traditional sparse methods will suffer. In this paper we focus on an alternative building block for sparse iterative solvers, the “matrix powers ker-nel” [x, Ax, $\rm A^{2}x$, …, $\rm A^{k}x$], and show that by organizing computations around this kernel, we can achieve near-minimal communication costs. We consider communication very broadly as both network communication in parallel code and memory hierarchy access in sequential code. In particular, we introduce a parallel algorithm for which the number of messages (total latency cost) is independent of the power k, and a sequential algorithm, that reduces both the number and volume of accesses, so that it is independent of k in both latency and bandwidth costs. This is part of a larger project to develop “communication-avoiding Krylov subspace methods,” which also addresses the numerical issues associated with these methods. Our algorithms work for general sparse matrices that “partition well”. We introduce parallel performance models of matrices arising from 2D and 3D problems and show predicted speedups over a conventional algorithm of up to $7x$ on a Petaflop-scale machine and up to $22x$ on computation across the Grid. Analogous sequential performance models of the same problems predict speedups over a conventional algorithm of up to $10x$ on an out-of-core implemen-tation, and up to $2.5x$ when we use our ideas to reduce off-chip latency and bandwidth to DRAM. Finally, we validate the model on an out-of-core sequential implementation and measured a speedup of over $3x$, which is close to the predicted speedup.

Parallel Approximate Matrix Factorization For Kernel Methods

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

An $N \log N$ Parallel Fast Direct Solver for Kernel Matrices

PBBFMM3D: A parallel black-box algorithm for kernel matrix-vector multiplication

Kronecker Factorization for Speeding Up Kernel Machines.

On the Parallel I/O Optimality of Linear Algebra Kernels: Near-Optimal Matrix Factorizations

Approximate Kernel Selection via Matrix Approximation

A High-Performance Parallel Algorithm for Nonnegative Matrix Factorization

Acceleration of Approximate Matrix Multiplications on GPUs

A hybrid stochastic interpolation and compression method for kernel matrices

The Fast Kernel Transform

Efficient and Scalable Kernel Matrix Approximations using Hierarchical Decomposition

On the Parallel I/O Optimality of Linear Algebra Kernels: Near-Optimal LU Factorization

On the Nystrom Approximation for Preconditioning in Kernel Machines

Kernel Approximation using Analog In-Memory Computing

A HOMOTOPY METHOD FOR MULTIKERNEL-BASED APPROXIMATION

Revisiting Memory Efficient Kernel Approximation: An Indefinite Learning Perspective

Computing Low-Rank Approximation of a Dense Matrix on Multicore CPUs with a GPU and Its Application to Solving a Hierarchically Semiseparable Linear System of Equations

Avoiding communication in sparse matrix computations

Iterative Kernel Regression with Preconditioning

Sparse Cholesky Factorization for Solving Nonlinear PDEs via Gaussian Processes