Abstract:In this paper we show how to accelerate randomized coordinate descent methods and achieve faster convergence rates without paying per-iteration costs in asymptotic running time. In particular, we show how to generalize and efficiently implement a method proposed by Nesterov, giving faster asymptotic running times for various algorithms that use standard coordinate descent as a black box. In addition to providing a proof of convergence for this new general method, we show that it is numerically stable, efficiently implementable, and in certain regimes, asymptotically optimal. To highlight the power of this algorithm, we show how it can used to create faster linear system solvers in several regimes: • We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. • We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. • We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. Beyond the independent interest of these solvers, we believe they highlight the versatility of the approach of this paper and we hope that they will open the door for further algorithmic improvements in the future.

Accelerating the Speed of Convergence for High-Order Methods to Solve Equations

On the Convergence of the Accelerated Riccati Iteration Method.

An extended study of two multistep higher order convergent methods for solving nonlinear equations

Larger convergence regions for an efficient two-step iterative method

Extended Efficient Multistep Solvers for Solving Equations in Banach Spaces

A One-Parameter Family of Methods with a Higher Order of Convergence for Equations in a Banach Space

Generalized Convergence for Multi-Step Schemes under Weak Conditions

Advancing convergence analysis: extending the scope of a sixth order method

Generalized high-order iterative methods for solutions of nonlinear systems and their applications

The Local Convergence of a Three-Step Sixth-Order Iterative Approach with the Basin of Attraction

Symmetric-Type Multi-Step Difference Methods for Solving Nonlinear Equations

Local convergence study of tenth-order iterative method in Banach spaces with basin of attraction

Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems

Accelerating iterative solvers via a two-dimensional minimum residual technique

New high-order convergence iteration methods without employing derivatives for solving nonlinear equations

A Nonlinear Acceleration Method for Iterative Algorithms

Symplectic Discretization Approach for Developing New Proximal Point Algorithm

New Family of Multi-Step Iterative Methods Based on Homotopy Perturbation Technique for Solving Nonlinear Equations

On Convergence Of A New Secant-Like Method For Solving Nonlinear Equations

Two-step inertial Bregman alternating structure-adapted proximal gradient descent algorithm for nonconvex and nonsmooth problems

Convergence of a Family of Methods with Symmetric, Antisymmetric Parameters and Weight Functions