Abstract:On current computer architectures, GMRES' performance can be limited by its communication cost to generate orthonormal basis vectors of the Krylov subspace. To address this performance bottleneck, its $s$-step variant orthogonalizes a block of $s$ basis vectors at a time, potentially reducing the communication cost by a factor of $s$. Unfortunately, for a large step size $s$, the solver can generate extremely ill-conditioned basis vectors, and to maintain stability in practice, a conservatively small step size is used, which limits the performance of the $s$-step solver. To enhance the performance using a small step size, in this paper, we introduce a two-stage block orthogonalization scheme. Similar to the original scheme, the first stage of the proposed method operates on a block of $s$ basis vectors at a time, but its objective is to maintain the well-conditioning of the generated basis vectors with a lower cost. The orthogonalization of the basis vectors is delayed until the second stage when enough basis vectors are generated to obtain higher performance. Our analysis shows the stability of the proposed two-stage scheme. The performance is improved because while the same amount of computation as the original scheme is required, most of the communication is done at the second stage of the proposed scheme, reducing the overall communication requirements. Our performance results with up to 192 NVIDIA V100 GPUs on the Summit supercomputer demonstrate that when solving a 2D Laplace problem, the two-stage approach can reduce the orthogonalization time and the total time-to-solution by the respective factors of up to $2.6\times$ and $1.6\times$ over the original $s$-step GMRES, which had already obtained the respective speedups of $2.1\times$ and $1.8\times$ over the standard GMRES. Similar speedups were obtained for 3D problems and for matrices from the SuiteSparse Matrix Collection.

A Nonlinear GMRES Optimization Algorithm for Canonical Tensor Decomposition

Randomized flexible GMRES with deflated restarting

Alternating Direction and Taylor Expansion Minimization Algorithms for Unconstrained Nuclear Norm Optimization.

Randomized sketched TT-GMRES for linear systems with tensor structure

Preconditioning Low Rank Generalized Minimal Residual Method (GMRES) for Implicit Discretizations of Matrix Differential Equations

Extended Gauss-Newton and ADMM-Gauss-Newton Algorithms for Low-Rank Matrix Optimization

On the Global Convergence of the Alternating Least Squares Method for Rank-One Approximation to Generic Tensors

Optimal Solutions of Well-Posed Linear Systems via Low-Precision Right-Preconditioned GMRES with Forward and Backward Stabilization

Preconditioned low-rank Riemannian optimization for linear systems with tensor product structure

Block GMRES Method with Inexact Breakdowns and Deflated Restarting

Restarted GMRES Method Augmented with the Combination of Harmonic Ritz Vectors and Error Approximations

Riemannian Optimization for Non-convex Euclidean Distance Geometry with Global Recovery Guarantees

Fast and Accurate Randomized Algorithms for Low-rank Tensor Decompositions

On IGMRES: an Incomplete Generalized Minimal Residual Method for Large Unsymmetric Linear Systems

Iterative Methods in GPU-Resident Linear Solvers for Nonlinear Constrained Optimization

Compressed Basis GMRES on High Performance GPUs

Improving the Performance of the GMRES Method using Mixed-Precision Techniques

GMRES with randomized sketching and deflated restarting

A numerically stable communication-avoiding s-step GMRES algorithm

Alternating minimization algorithms for graph regularized tensor completion

Two-Stage Block Orthogonalization to Improve Performance of $s$-step GMRES