Abstract:In this paper we show how to accelerate randomized coordinate descent methods and achieve faster convergence rates without paying per-iteration costs in asymptotic running time. In particular, we show how to generalize and efficiently implement a method proposed by Nesterov, giving faster asymptotic running times for various algorithms that use standard coordinate descent as a black box. In addition to providing a proof of convergence for this new general method, we show that it is numerically stable, efficiently implementable, and in certain regimes, asymptotically optimal. To highlight the power of this algorithm, we show how it can used to create faster linear system solvers in several regimes: • We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. • We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. • We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. Beyond the independent interest of these solvers, we believe they highlight the versatility of the approach of this paper and we hope that they will open the door for further algorithmic improvements in the future.

Fast Nonoverlapping Block Jacobi Method for the Dual Rudin--Osher--Fatemi Model

Augmented Lagrangian Method, Dual Methods, and Split Bregman Iteration for ROF, Vectorial TV, and High Order Models

Randomized block coordinate descent method for linear ill-posed problems

Fast X-ray CT image reconstruction using the linearized augmented Lagrangian method with ordered subsets

Constrained Block-Iterative Landweber Scheme for Image Reconstruction

Nesterov's Accelerated Jacobi-Type Methods for Large-scale Symmetric Positive Semidefinite Linear Systems

On the alternating randomized block Kaczmarz method

A block Lanczos method for large-scale quadratic minimization problems with orthogonality constraints

Fast Iterative Solution of Multiple Right-Hand Sides MoM Linear Systems on CPUs and GPUs Computers

Fast block Krylov subspace methods for solving sequences of dense MoM linear systems with multiple right-hand sides

A distributed Douglas-Rachford splitting method for solving linear constrained multi-block weakly convex problems

A Globally Convergent Distributed Jacobi Scheme for Block-Structured Nonconvex Constrained Optimization Problems

Deep Block Proximal Linearised Minimisation Algorithm for Non-convex Inverse Problems

Convergence Analysis of a Finite Difference Scheme for the Gradient Flow associated with the ROF Model

A spectrally preconditioned and initially deflated variant of the restarted block GMRES method for solving multiple right-hand sides linear systems

On a Fast Deterministic Block Kaczmarz Method for Solving Large-Scale Linear Systems.

Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems

Orthogonal Block Kaczmarz Algorithm Based on Preprocessing Technology

On a Nonlinear Fast Deterministic Block Kaczmarz Method for Solving Nonlinear Equations

An Inexact Dual Alternating Direction Method of Multipliers for Image Decomposition and Restoration

Block Broyden's Methods for Solving Nonlinear Equations.