Abstract:In this paper we show how to accelerate randomized coordinate descent methods and achieve faster convergence rates without paying per-iteration costs in asymptotic running time. In particular, we show how to generalize and efficiently implement a method proposed by Nesterov, giving faster asymptotic running times for various algorithms that use standard coordinate descent as a black box. In addition to providing a proof of convergence for this new general method, we show that it is numerically stable, efficiently implementable, and in certain regimes, asymptotically optimal. To highlight the power of this algorithm, we show how it can used to create faster linear system solvers in several regimes: • We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. • We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. • We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. Beyond the independent interest of these solvers, we believe they highlight the versatility of the approach of this paper and we hope that they will open the door for further algorithmic improvements in the future.

Improved Seed Methods for Symmetric Positive Definite Linear Equations with Multiple Right-hand Sides

Lanczos Seed Projection Methods for Solving Symmetric Shifted System with Multiple Right-hand Sides

A semi-conjugate gradient method for solving unsymmetric positive definite linear systems

Deflated Iterative Methods for Linear Equations with Multiple Right-Hand Sides

Conjugate Residual Squared Method and Its Improvement for Non-Symmetric Linear Systems.

Iterative Methods for Sparse Symmetric Multilinear Systems

Restrictive Preconditioners for Conjugate Gradient Methods for Symmetric Positive Definite Linear Systems

Conjugate Direction Methods Under Inconsistent Systems

An Enhancement of the Accuracy of the BiCGStab Method for Solving Linear Systems with Single or Multiple Right-Hand Sides

An Efficient Modified Residual-Based Algorithm for Large Scale Symmetric Nonlinear Equations by Approximating Successive Iterated Gradients

Computing Symmetric Positive Definite Solutions of Three Types of Nonlinear Matrix Equations

Iterative Methods by Space Decomposition and Subspace Correction

Preconditioned Low-Rank Riemannian Optimization for Symmetric Positive Definite Linear Matrix Equations

Modified CRI iteration methods for complex symmetric indefinite linear systems

Nesterov's Accelerated Jacobi-Type Methods for Large-scale Symmetric Positive Semidefinite Linear Systems

A sufficient descent three-term conjugate gradient method via symmetric rank-one update for large-scale optimization

Preconditioning Technique for Symmetric M-matrices

Preconditioned iterative method for nonsymmetric saddle point linear systems

Iterative Solution of Multi-Shifted Linear Systems with Multiple Right-Hand Sides.

Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems

On HSS and AHSS Iteration Methods for Nonsymmetric Positive Definite Toeplitz Systems.