Abstract:While effective in practice, iterative methods for solving large systems of linear equations can be significantly affected by problem-dependent condition number quantities. This makes characterizing their time complexity challenging, particularly when we wish to make comparisons between deterministic and stochastic methods, that may or may not rely on preconditioning and/or fast matrix multiplication. In this work, we consider a fine-grained notion of complexity for iterative linear solvers which we call the spectral tail condition number, $\kappa_\ell$, defined as the ratio between the $\ell$th largest and the smallest singular value of the matrix representing the system. Concretely, we prove the following main algorithmic result: Given an $n\times n$ matrix $A$ and a vector $b$, we can find $\tilde{x}$ such that $\|A\tilde{x}-b\|\leq\epsilon\|b\|$ in time $\tilde{O}(\kappa_\ell\cdot n^2\log 1/\epsilon)$ for any $\ell = O(n^{\frac1{\omega-1}})=O(n^{0.729})$, where $\omega \approx 2.372$ is the current fast matrix multiplication exponent. This guarantee is achieved by Sketch-and-Project with Nesterov's acceleration. Some of the implications of our result, and of the use of $\kappa_\ell$, include direct improvement over a fine-grained analysis of the Conjugate Gradient method, suggesting a stronger separation between deterministic and stochastic iterative solvers; and relating the complexity of iterative solvers to the ongoing algorithmic advances in fast matrix multiplication, since the bound on $\ell$ improves with $\omega$. Our main technical contributions are new sharp characterizations for the first and second moments of the random projection matrix that commonly arises in sketching algorithms, building on a combination of techniques from combinatorial sampling via determinantal point processes and Gaussian universality results from random matrix theory.

Numerical tolerance for spectral decompositions of random matrices

Algorithms that satisfy a stopping criterion, probably

Noisy matrix decomposition via convex relaxation: Optimal rates in high dimensions

On the Numerical Approximation of the Karhunen-Loève Expansion for Random Fields with Random Discrete Data

Entrywise error bounds for low-rank approximations of kernel matrices

Approximating matrix eigenvalues by subspace iteration with repeated random sparsification

A general error analysis for randomized low-rank approximation with application to data assimilation

Fine-grained Analysis and Faster Algorithms for Iteratively Solving Linear Systems

Efficient error and variance estimation for randomized matrix computations

Spectral Condition-Number Estimation of Large Sparse Matrices

Analysis of eigenvalue condition numbers for a class of randomized numerical methods for singular matrix pencils

Perturbation Analysis of Randomized SVD and its Applications to High-dimensional Statistics

Sublinear Time Eigenvalue Approximation via Random Sampling

Numerical accuracy of the derivative-expansion-based functional renormalization group

Optimal Lower Bound on Eigenvector Overlaps for non-Hermitian Random Matrices

Optimal Randomized Approximations for Matrix-Based Rényi’s Entropy

High Order Numerical Methods To Approximate The Singular Value Decomposition

Randomized estimation of spectral densities of large matrices made accurate

Efficient Computation of Limit Spectra of Sample Covariance Matrices

Theory and numerics of subspace approximation of eigenvalue problems

Ill-Conditioning Of The Truncated Singular Value Decomposition, Tikhonov Regularization And Their Applications To Numerical Partial Differential Equations