Abstract:In this paper we show how to accelerate randomized coordinate descent methods and achieve faster convergence rates without paying per-iteration costs in asymptotic running time. In particular, we show how to generalize and efficiently implement a method proposed by Nesterov, giving faster asymptotic running times for various algorithms that use standard coordinate descent as a black box. In addition to providing a proof of convergence for this new general method, we show that it is numerically stable, efficiently implementable, and in certain regimes, asymptotically optimal. To highlight the power of this algorithm, we show how it can used to create faster linear system solvers in several regimes: • We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. • We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. • We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. Beyond the independent interest of these solvers, we believe they highlight the versatility of the approach of this paper and we hope that they will open the door for further algorithmic improvements in the future.

The accelerated tensor Kaczmarz algorithm with adaptive parameters for solving tensor systems

Randomized Average Kaczmarz Algorithm for Tensor Linear Systems

Randomized extended average block Kaczmarz method for inconsistent tensor equations under t-product

Randomized Kaczmarz for tensor linear systems

An Accelerated Block Randomized Kaczmarz Method

Solving the system of nonsingular tensor equations via randomized Kaczmarz-like method

Randomized sketched TT-GMRES for linear systems with tensor structure

Randomized Kaczmarz methods for tensor complementarity problems

Block Matrix and Tensor Randomized Kaczmarz Methods for Linear Feasibility Problems

Randomized regularized extended Kaczmarz algorithms for tensor recovery

Faster randomized block sparse Kaczmarz by averaging

Preasymptotic convergence of randomized Kaczmarz method

Accelerated Kaczmarz Algorithms using History Information

Randomized algorithms for Kroncecker tensor decomposition and applications

Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems

On global randomized block Kaczmarz algorithm for solving large-scale matrix equations

Randomized Kaczmarz method with adaptive stepsizes for inconsistent linear systems

Regularized Kaczmarz Algorithms for Tensor Recovery

Acceleration of Randomized Kaczmarz Method via the Johnson-Lindenstrauss Lemma

On the adaptive deterministic block Kaczmarz method with momentum for solving large-scale consistent linear systems