Abstract:In this paper we show how to accelerate randomized coordinate descent methods and achieve faster convergence rates without paying per-iteration costs in asymptotic running time. In particular, we show how to generalize and efficiently implement a method proposed by Nesterov, giving faster asymptotic running times for various algorithms that use standard coordinate descent as a black box. In addition to providing a proof of convergence for this new general method, we show that it is numerically stable, efficiently implementable, and in certain regimes, asymptotically optimal. To highlight the power of this algorithm, we show how it can used to create faster linear system solvers in several regimes: • We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. • We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. • We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. Beyond the independent interest of these solvers, we believe they highlight the versatility of the approach of this paper and we hope that they will open the door for further algorithmic improvements in the future.

Algorithms for descent along nodal straight lines in the problem of estimating regression equations using the least absolute deviations method

New algorithms for computing the least trimmed squares estimator

Least Squares Solution with the Minimum-Norm to General Matrix Equations Via Iteration.

A Direction Optimization Least Mean Square Algorithm

Linearly convergent adjoint free solution of least squares problems by random descent

Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems

A Fast Block Coordinate Descent Method for Solving Linear Least-Squares Problems

Iterative algorithm for minimal norm least squares solution to general linear matrix equations

A new approximate descent derivative-free algorithm for large-scale nonlinear symmetric equations

Analytic analysis of the worst-case complexity of the gradient method with exact line search and the Polyak stepsize

Fast Algorithms for lp-Regression

A Rapidly Convergent Iterative Method for the Solution of the Generalised Nonlinear Least Squares Problem

Optimisation of Least Squares Algorithm: A Study of Frame Based Programming Techniques in Horizontal Networks

The discrete analogue of the method of quickest descent for an inverse acoustic problem in case of a smooth source

Variations on least squares

On adapting Nesterov's scheme to accelerate iterative methods for linear problems

Gradient-based Sampling: An Adaptive Importance Sampling for Least-squares

Network flows that solve least squares for linear equations

Accelerated SGD for Non-Strongly-Convex Least Squares

Fast Gradient Method for Low-Rank Matrix Estimation

Semi-Smooth Newton Algorithm for Non-Convex Penalized Linear Regression