Abstract:While effective in practice, iterative methods for solving large systems of linear equations can be significantly affected by problem-dependent condition number quantities. This makes characterizing their time complexity challenging, particularly when we wish to make comparisons between deterministic and stochastic methods, that may or may not rely on preconditioning and/or fast matrix multiplication. In this work, we consider a fine-grained notion of complexity for iterative linear solvers which we call the spectral tail condition number, $\kappa_\ell$, defined as the ratio between the $\ell$th largest and the smallest singular value of the matrix representing the system. Concretely, we prove the following main algorithmic result: Given an $n\times n$ matrix $A$ and a vector $b$, we can find $\tilde{x}$ such that $\|A\tilde{x}-b\|\leq\epsilon\|b\|$ in time $\tilde{O}(\kappa_\ell\cdot n^2\log 1/\epsilon)$ for any $\ell = O(n^{\frac1{\omega-1}})=O(n^{0.729})$, where $\omega \approx 2.372$ is the current fast matrix multiplication exponent. This guarantee is achieved by Sketch-and-Project with Nesterov's acceleration. Some of the implications of our result, and of the use of $\kappa_\ell$, include direct improvement over a fine-grained analysis of the Conjugate Gradient method, suggesting a stronger separation between deterministic and stochastic iterative solvers; and relating the complexity of iterative solvers to the ongoing algorithmic advances in fast matrix multiplication, since the bound on $\ell$ improves with $\omega$. Our main technical contributions are new sharp characterizations for the first and second moments of the random projection matrix that commonly arises in sketching algorithms, building on a combination of techniques from combinatorial sampling via determinantal point processes and Gaussian universality results from random matrix theory.

On $O(n)$ Algorithms for Projection onto the Top-$k$-sum Sublevel Set

O(log T) Projections for Stochastic Optimization of Smooth and Strongly Convex Functions

Proving Inequalities and Solving Global Optimization Problems Via Simplified CAD Projection

Convergence Rate Analysis of a Dykstra-Type Projection Algorithm

Derivative-Free Alternating Projection Algorithms for General Nonconvex-Concave Minimax Problems

Projection-Free Variance Reduction Methods for Stochastic Constrained Multi-Level Compositional Optimization

Computing the Projection on the Intersection of Linear Equality and Cardinality Constraints

On the Iteration Complexity of Some Projection Methods for Monotone Linear Variational Inequalities

Fine-grained Analysis and Faster Algorithms for Iteratively Solving Linear Systems

A Projection-Free Method for Solving Convex Bilevel Optimization Problems

Successive Projection for Solving Systems of Nonlinear Equations/Inequalities

Most Iterations of Projections Converge

A General Analysis Framework of Lower Complexity Bounds for Finite-Sum Optimization

An Infeasible-Point Subgradient Method Using Adaptive Approximate Projections

Single-Projection Procedure for Infinite Dimensional Convex Optimization Problems

Projection-Free Non-Smooth Convex Programming

Semismooth Newton Algorithm for Efficient Projections onto $\ell_1, ∞$-norm Ball

The splitting algorithms by Ryu and by Malitsky-Tam applied to normal cones of linear subspaces converge strongly to the projection onto the intersection

The splitting algorithms by Ryu, by Malitsky-Tam, and by Campoy applied to normal cones of linear subspaces converge strongly to the projection onto the intersection