Abstract:In this paper we show how to accelerate randomized coordinate descent methods and achieve faster convergence rates without paying per-iteration costs in asymptotic running time. In particular, we show how to generalize and efficiently implement a method proposed by Nesterov, giving faster asymptotic running times for various algorithms that use standard coordinate descent as a black box. In addition to providing a proof of convergence for this new general method, we show that it is numerically stable, efficiently implementable, and in certain regimes, asymptotically optimal. To highlight the power of this algorithm, we show how it can used to create faster linear system solvers in several regimes: • We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. • We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. • We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. Beyond the independent interest of these solvers, we believe they highlight the versatility of the approach of this paper and we hope that they will open the door for further algorithmic improvements in the future.

The APC Algorithm of Solving Large-Scale Linear Systems: A Generalized Analysis

Distributed Accelerated Projection-Based Consensus Decomposition

Scalable, Distributed Algorithms for Solving Linear Equations via Double-Layered Networks

A Networked Parallel Algorithm for Solving Linear Algebraic Equations

A Non-Krylov subspace Method for Solving Large and Sparse Linear System of Equations

A Resilient Distributed Algorithm for Solving Linear Equations

Distributed Linear Equations over Random Networks

Accelerating Convergence by Augmented Rayleigh-Ritz Projections for Large-Scale Eigenpair Computation

On the Effects of Data Heterogeneity on the Convergence Rates of Distributed Linear System Solvers

Projection Consensus for Solving Linear Equations over Random Networks

Scalable Distributed Least Square Algorithms for Large-Scale Linear Equations Via an Optimization Approach

Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems

Conjugate Residual Squared Method and Its Improvement for Non-Symmetric Linear Systems.

Distributed Computing for Huge-Scale Linear Programming

Successive Projection for Solving Systems of Nonlinear Equations/Inequalities

Projection-based Prediction-Correction Method for Distributed Consensus Optimization

Distributed algorithms of solving linear matrix equations via double-layered networks

Incremental Large-Scale Electrostatic Analysis

A Novel Greedy Kaczmarz Method For Solving Consistent Linear Systems

Generalized Block Lanczos Methods for Large Unsymmetric Eigenproblems