Abstract:In this paper we show how to accelerate randomized coordinate descent methods and achieve faster convergence rates without paying per-iteration costs in asymptotic running time. In particular, we show how to generalize and efficiently implement a method proposed by Nesterov, giving faster asymptotic running times for various algorithms that use standard coordinate descent as a black box. In addition to providing a proof of convergence for this new general method, we show that it is numerically stable, efficiently implementable, and in certain regimes, asymptotically optimal. To highlight the power of this algorithm, we show how it can used to create faster linear system solvers in several regimes: • We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. • We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. • We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. Beyond the independent interest of these solvers, we believe they highlight the versatility of the approach of this paper and we hope that they will open the door for further algorithmic improvements in the future.

Fast and Accurate Randomized Algorithms for Linear Systems and Eigenvalue Problems

Fast and Forward Stable Randomized Algorithms for Linear Least-Squares Problems

A Fast Randomized Algorithm for Computing an Approximate Null Space

Randomized Iterative Methods for Linear Systems

Randomized numerical linear algebra: Foundations and algorithms

Randomized algorithms for low-rank matrix approximation: Design, analysis, and applications

Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems

Iterative methods for solving factorized linear systems

Randomized methods for rank-deficient linear systems

Fine-grained Analysis and Faster Algorithms for Iteratively Solving Linear Systems

Randomized sketching of nonlinear eigenvalue problems

Speeding Up Krylov Subspace Methods for Computing [math] via Randomization

Solving Dense Linear Systems Faster Than via Preconditioning

Fast randomized least-squares solvers can be just as accurate and stable as classical direct solvers

Efficient Bounds and Estimates for Canonical Angles in Randomized Subspace Approximations

Pseudoinverse-free randomized block iterative algorithms for consistent and inconsistent linear systems

Approximating matrix eigenvalues by subspace iteration with repeated random sparsification

Sketch 'n Solve: An Efficient Python Package for Large-Scale Least Squares Using Randomized Numerical Linear Algebra

Efficient Randomized Algorithms for the Fixed-Precision Low-Rank Matrix Approximation