Abstract:In this paper we show how to accelerate randomized coordinate descent methods and achieve faster convergence rates without paying per-iteration costs in asymptotic running time. In particular, we show how to generalize and efficiently implement a method proposed by Nesterov, giving faster asymptotic running times for various algorithms that use standard coordinate descent as a black box. In addition to providing a proof of convergence for this new general method, we show that it is numerically stable, efficiently implementable, and in certain regimes, asymptotically optimal. To highlight the power of this algorithm, we show how it can used to create faster linear system solvers in several regimes: • We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. • We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. • We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. Beyond the independent interest of these solvers, we believe they highlight the versatility of the approach of this paper and we hope that they will open the door for further algorithmic improvements in the future.

Randomized Methods for Linear Constraints: Convergence Rates and Conditioning

Randomized Iterative Methods for Linear Systems

Convergence properties of the randomized extended Gauss-Seidel and Kaczmarz methods

Randomized Kaczmarz iteration methods: Algorithmic extensions and convergence theory

Linear Convergence of Stochastic Iterative Greedy Algorithms With Sparse Constraints

Speeding up Linear Programming using Randomized Linear Algebra

Randomized sketch descent methods for non-separable linearly constrained optimization

Acceleration of Randomized Kaczmarz Method via the Johnson-Lindenstrauss Lemma

Randomized methods for rank-deficient linear systems

Random Multi-Constraint Projection: Stochastic Gradient Methods for Convex Optimization with Many Constraints

Randomized Kaczmarz Algorithm for Inconsistent Linear Systems: An Exact MSE Analysis

Randomized Kaczmarz method with adaptive stepsizes for inconsistent linear systems

Linearly convergent adjoint free solution of least squares problems by random descent

Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems

Faster Randomized Methods for Orthogonality Constrained Problems

Convergence Rates for Greedy Kaczmarz Algorithms, and Faster Randomized Kaczmarz Rules Using the Orthogonality Graph

Linear Phase Transition in Random Linear Constraint Satisfaction Problem

Successive Projection for Solving Systems of Nonlinear Equations/Inequalities

Deterministic Versus Randomized Kaczmarz Iterative Projection

Regularized randomized iterative algorithms for factorized linear systems

Randomized Block Adaptive Linear System Solvers