Abstract:In this paper we show how to accelerate randomized coordinate descent methods and achieve faster convergence rates without paying per-iteration costs in asymptotic running time. In particular, we show how to generalize and efficiently implement a method proposed by Nesterov, giving faster asymptotic running times for various algorithms that use standard coordinate descent as a black box. In addition to providing a proof of convergence for this new general method, we show that it is numerically stable, efficiently implementable, and in certain regimes, asymptotically optimal. To highlight the power of this algorithm, we show how it can used to create faster linear system solvers in several regimes: • We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. • We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. • We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. Beyond the independent interest of these solvers, we believe they highlight the versatility of the approach of this paper and we hope that they will open the door for further algorithmic improvements in the future.

Coordinate Descent for MCP/SCAD Penalized Least Squares Converges Linearly

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Efficiency of Coordinate Descent Methods For Structured Nonconvex Optimization

Block Coordinate Descent Methods for Structured Nonconvex Optimization with Nonseparable Constraints: Optimality Conditions and Global Convergence

A Semismooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning

Coordinate Descent Algorithm for Ramp Loss Linear Programming Support Vector Machines

Semi-Smooth Newton Algorithm for Non-Convex Penalized Linear Regression

Feature Clustering for Accelerating Parallel Coordinate Descent

A Fast Coordinate Descent Method for High-Dimensional Non-Negative Least Squares using a Unified Sparse Regression Framework

A randomized sparse Kaczmarz solver for sparse signal recovery via minimax‐concave penalty

A Cyclic Coordinate Descent Method for Convex Optimization on Polytopes

A Simple and Fast Coordinate-Descent Augmented-Lagrangian Solver for Model Predictive Control

Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation

A Concave Conjugate Approach for Nonconvex Penalized Regression with the MCP Penalty.

Semismooth Newton Coordinate Descent Algorithm for Elastic-Net Penalized Huber Loss Regression and Quantile Regression

Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems

Dykstra's Algorithm, ADMM, and Coordinate Descent: Connections, Insights, and Extensions

Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation.

Stochastic Coordinate Descent Methods for Regularized Smooth and Nonsmooth Losses.

A new penalized least absolute deviation model for high dimensional sparse linear regression and an efficient sequential linear programming algorithm