Abstract:We present a new class of preconditioned iterative methods for solving linear systems of the form $Ax = b$. Our methods are based on constructing a low-rank Nyström approximation to $A$ using sparse random sketching. This approximation is used to construct a preconditioner, which itself is inverted quickly using additional levels of random sketching and preconditioning. We prove that the convergence of our methods depends on a natural average condition number of $A$, which improves as the rank of the Nyström approximation increases. Concretely, this allows us to obtain faster runtimes for a number of fundamental linear algebraic problems: 1. We show how to solve any $n\times n$ linear system that is well-conditioned except for $k$ outlying large singular values in $\tilde{O}(n^{2.065} + k^\omega)$ time, improving on a recent result of [Dereziński, Yang, STOC 2024] for all $k \gtrsim n^{0.78}$. 2. We give the first $\tilde{O}(n^2 + {d_\lambda}^{\omega}$) time algorithm for solving a regularized linear system $(A + \lambda I)x = b$, where $A$ is positive semidefinite with effective dimension $d_\lambda$. This problem arises in applications like Gaussian process regression. 3. We give faster algorithms for approximating Schatten $p$-norms and other matrix norms. For example, for the Schatten 1 (nuclear) norm, we give an algorithm that runs in $\tilde{O}(n^{2.11})$ time, improving on an $\tilde{O}(n^{2.18})$ method of [Musco et al., ITCS 2018]. Interestingly, previous state-of-the-art algorithms for most of the problems above relied on stochastic iterative methods, like stochastic coordinate and gradient descent. Our work takes a completely different approach, instead leveraging tools from matrix sketching.

Preconditioner Design via Bregman Divergences

Preconditioner Design via the Bregman Divergence

A New Matrix Truncation Method for Improving Approximate Factorisation Preconditioners

Matrix Nearness Problems with Off-Block-Diagonal Rank Constraints

Preconditioning of spectral methods via Birkhoff interpolation

Two-level Nyström-Schur preconditioner for sparse symmetric positive definite matrices

A Modified SSOR-like Preconditioner for Non-Hermitian Positive Definite Matrices

Faster Linear Systems and Matrix Norm Approximation via Multi-level Sketched Preconditioning

Scalable Approximate Optimal Diagonal Preconditioning

New Preconditioners for Nonsymmetric and Indefinite Saddle Point Linear Systems

Constraint Preconditioners for Symmetric Indefinite Matrices

A Class Of Preconditioners For Non-Hermitian Positive Definite Linear Systems

Generalized Tridiagonal Preconditioners for Solving Linear Systems

A new preconditioner for indefinite and asymmetric matrices

Compact quasi‐Newton preconditioners for symmetric positive definite linear systems

A Class of Approximate Inverse Preconditioners for Solving Linear Systems

Preconditioned Low-Rank Riemannian Optimization for Symmetric Positive Definite Linear Matrix Equations

Hermitian Preconditioning for a class of Non-Hermitian Linear Systems

Preconditioning Low Rank Generalized Minimal Residual Method (GMRES) for Implicit Discretizations of Matrix Differential Equations

A Class of Preconditioners Based on Symmetric-Triangular Decomposition and Matrix Splitting for Generalized Saddle Point Problems

A universal matrix-free split preconditioner for the fixed-point iterative solution of non-symmetric linear systems