Abstract:The performance of sparse iterative solvers is typically limited by sparse matrix-vector multiplication, which is itself limited by memory system and network performance. As the gap between computation and communication speed continues to widen, these traditional sparse methods will suffer. In this paper we focus on an alternative building block for sparse iterative solvers, the “matrix powers ker-nel” [x, Ax, $\rm A^{2}x$, …, $\rm A^{k}x$], and show that by organizing computations around this kernel, we can achieve near-minimal communication costs. We consider communication very broadly as both network communication in parallel code and memory hierarchy access in sequential code. In particular, we introduce a parallel algorithm for which the number of messages (total latency cost) is independent of the power k, and a sequential algorithm, that reduces both the number and volume of accesses, so that it is independent of k in both latency and bandwidth costs. This is part of a larger project to develop “communication-avoiding Krylov subspace methods,” which also addresses the numerical issues associated with these methods. Our algorithms work for general sparse matrices that “partition well”. We introduce parallel performance models of matrices arising from 2D and 3D problems and show predicted speedups over a conventional algorithm of up to $7x$ on a Petaflop-scale machine and up to $22x$ on computation across the Grid. Analogous sequential performance models of the same problems predict speedups over a conventional algorithm of up to $10x$ on an out-of-core implemen-tation, and up to $2.5x$ when we use our ideas to reduce off-chip latency and bandwidth to DRAM. Finally, we validate the model on an out-of-core sequential implementation and measured a speedup of over $3x$, which is close to the predicted speedup.

Parallel and memory-efficient realization of KSP/KNN algorithm

A Parallel Simulator for Massive Reservoir Models Utilizing Distributed-Memory Parallel Systems

Parallel Data Partitioning Strategy In Solving Large Scale Electromagnetic Scattering Problems

Efficient Parallel Processing of High-Dimensional Spatial K NN Queries

$\Sf {simknn}$: A Scalable Method for In-Memoryknn Search over Moving Objects in Road Networks.

Memory-Efficient RkNN Retrieval by Nonlinear k-Distance Approximation

Parallel Construction of Multiresolution Representation for Massive Meshes Based on External Memory Octree

Efficient parallel processing for K-nearest-neighbor search in spatial databases

MPR — A Partitioning-Replication Framework for Multi-Processing kNN Search on Road Networks

A Parallel Method With Hybrid Algorithms For Mixed Integer Nonlinear Programming

Efficient Algorithms For Historical Continuous Knn Query Processing Over Moving Object Trajectories

Parallelization of Module Network Structure Learning and Performance Tuning on SMP

Contorting High Dimensional Data for Efficient Main Memory KNN Processing

Parallel Network RAM: Effectively Utilizing Global Cluster Memory for Large Data-Intensive Parallel Programs

Efficient Nearest Neighbor Search Using Dynamic Programming

Shared Memory Parallelization of MTTKRP for Dense Tensors

Avoiding communication in sparse matrix computations

Design and implementation of a parallel geographically weighted k-nearest neighbor classifier

Processing-In-Hierarchical-Memory Architecture for Billion-Scale Approximate Nearest Neighbor Search

CkNN monitoring based on parallel pre-computing

Efficient Probabilistic K-NN Computation in Uncertain Sensor Networks