Abstract:In this paper we show how to accelerate randomized coordinate descent methods and achieve faster convergence rates without paying per-iteration costs in asymptotic running time. In particular, we show how to generalize and efficiently implement a method proposed by Nesterov, giving faster asymptotic running times for various algorithms that use standard coordinate descent as a black box. In addition to providing a proof of convergence for this new general method, we show that it is numerically stable, efficiently implementable, and in certain regimes, asymptotically optimal. To highlight the power of this algorithm, we show how it can used to create faster linear system solvers in several regimes: • We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. • We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. • We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. Beyond the independent interest of these solvers, we believe they highlight the versatility of the approach of this paper and we hope that they will open the door for further algorithmic improvements in the future.

An Efficient Modified Residual-Based Algorithm for Large Scale Symmetric Nonlinear Equations by Approximating Successive Iterated Gradients

A new approximate descent derivative-free algorithm for large-scale nonlinear symmetric equations

Two Modified Single-Parameter Scaling Broyden–Fletcher–Goldfarb–Shanno Algorithms for Solving Nonlinear System of Symmetric Equations

A Modified Nonlinear Conjugate Gradient Algorithm for Large-Scale Nonsmooth Convex Optimization

Convergence Properties of an Iterative Method for Solving Symmetric Non-linear Equations

A Modified Spectral PRP Conjugate Gradient Projection Method for Solving Large-Scale Monotone Equations and Its Application in Compressed Sensing

A New Hybrid Descent Algorithm for Large-Scale Nonconvex Optimization and Application to Some Image Restoration Problems

A derivative-free memoryless BFGS hyperplane projection method for solving large-scale nonlinear monotone equations

Efficient Numerical Algorithm for Large-Scale Damped Natural Gradient Descent

Nonlinearly preconditioned semismooth Newton algorithms for nonlinear nonsmooth systems

An Efficient Iterative Approach to Large Sparse Nonlinear Systems with Non-Hermitian Jacobian Matrices

A structured diagonal Hessian approximation method with evaluation complexity analysis for nonlinear least squares

A residual-based weighted nonlinear Kaczmarz method for solving nonlinear systems of equations

Incremental Gauss--Newton Methods with Superlinear Convergence Rates

Two Descent Dai-Yuan Conjugate Gradient Methods for Systems of Monotone Nonlinear Equations

A New Three-Term Spectral Conjugate Gradient Algorithm with Higher Numerical Performance for Solving Large Scale Optimization Problems Based on Quasi-Newton Equation.

A novel greedy Gauss-Seidel method for solving large linear least squares problem

A Robust High-Order Residual Distribution Type Scheme for Steady Euler Equations on Unstructured Grids

Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems

A Novel Joint Diagonalization Approach for Linear Stochastic Systems and Reliability Analysis

Online Learning Guided Quasi-Newton Methods with Global Non-Asymptotic Convergence