Abstract:We present a new class of preconditioned iterative methods for solving linear systems of the form $Ax = b$. Our methods are based on constructing a low-rank Nyström approximation to $A$ using sparse random sketching. This approximation is used to construct a preconditioner, which itself is inverted quickly using additional levels of random sketching and preconditioning. We prove that the convergence of our methods depends on a natural average condition number of $A$, which improves as the rank of the Nyström approximation increases. Concretely, this allows us to obtain faster runtimes for a number of fundamental linear algebraic problems: 1. We show how to solve any $n\times n$ linear system that is well-conditioned except for $k$ outlying large singular values in $\tilde{O}(n^{2.065} + k^\omega)$ time, improving on a recent result of [Dereziński, Yang, STOC 2024] for all $k \gtrsim n^{0.78}$. 2. We give the first $\tilde{O}(n^2 + {d_\lambda}^{\omega}$) time algorithm for solving a regularized linear system $(A + \lambda I)x = b$, where $A$ is positive semidefinite with effective dimension $d_\lambda$. This problem arises in applications like Gaussian process regression. 3. We give faster algorithms for approximating Schatten $p$-norms and other matrix norms. For example, for the Schatten 1 (nuclear) norm, we give an algorithm that runs in $\tilde{O}(n^{2.11})$ time, improving on an $\tilde{O}(n^{2.18})$ method of [Musco et al., ITCS 2018]. Interestingly, previous state-of-the-art algorithms for most of the problems above relied on stochastic iterative methods, like stochastic coordinate and gradient descent. Our work takes a completely different approach, instead leveraging tools from matrix sketching.

PRECONDITION TECHNIQUES FOR ACCELERATED LINEARIZED BREGMAN ALGORITHMS

A preconditioned alternating minimization framework for nonconvex and half quadratic regularization

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

A cut-and-project perspective for linearized Bregman iterations

Bregman Augmented Lagrangian and Its Acceleration

An Accelerated Linearly Convergent Stochastic L-BFGS Algorithm

DeepOnet Based Preconditioning Strategies For Solving Parametric Linear Systems of Equations

Accelerated Bregman gradient methods for relatively smooth and relatively Lipschitz continuous minimization problems

Auto-conditioned primal-dual hybrid gradient method and alternating direction method of multipliers

Large Scale Constrained Linear Regression Revisited: Faster Algorithms via Preconditioning

Efficient Acceleration Framework for Complex‐Valued Linear Systems: Alternating Anderson Accelerating Preconditioned Richardson Approach

Faster Linear Systems and Matrix Norm Approximation via Multi-level Sketched Preconditioning

Delayed Projection Techniques for Linearly Constrained Problems: Convergence Rates, Acceleration, and Applications

A Bregman Proximal Stochastic Gradient Method with Extrapolation for Nonconvex Nonsmooth Problems

Acceleration and restart for the randomized Bregman-Kaczmarz method

A Bregman-Kaczmarz method for nonlinear systems of equations

Accelerated Bregman Proximal Gradient Methods for Relatively Smooth Convex Optimization

Asynchronous Algorithms for Solving Linear Programs

Solving Dense Linear Systems Faster Than via Preconditioning

Fast Linearized Bregman Iteration for Compressive Sensing and Sparse Denoising

Preconditioned golden ratio primal-dual algorithm with linesearch