Abstract:In this paper we show how to accelerate randomized coordinate descent methods and achieve faster convergence rates without paying per-iteration costs in asymptotic running time. In particular, we show how to generalize and efficiently implement a method proposed by Nesterov, giving faster asymptotic running times for various algorithms that use standard coordinate descent as a black box. In addition to providing a proof of convergence for this new general method, we show that it is numerically stable, efficiently implementable, and in certain regimes, asymptotically optimal. To highlight the power of this algorithm, we show how it can used to create faster linear system solvers in several regimes: • We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. • We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. • We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. Beyond the independent interest of these solvers, we believe they highlight the versatility of the approach of this paper and we hope that they will open the door for further algorithmic improvements in the future.

Efficient Algorithms and Error Analysis for the Modified Nystrom Method.

Improving the Modified Nyström Method Using Spectral Shifting

Improving CUR Matrix Decomposition and the Nyström Approximation Via Adaptive Sampling

SPSD Matrix Approximation Vis Column Selection: Theories, Algorithms, and Extensions

Improved Bounds for the Nyström Method with Application to Kernel Classification

Making the Nyström method highly accurate for low-rank approximations

A generalized Nystrom method with column sketching for low-rank approximation of nonsymmetric matrices

Towards More Efficient SPSD Matrix Approximation and CUR Matrix Decomposition

Nyström Subsampling Method for Coefficient-Based Regularized Regression

Non-uniform Nyström approximation for sparse kernel regression: Theoretical analysis and experimental evaluation

Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems

Column and row subset selection using nuclear scores: algorithms and theory for Nyström approximation, CUR decomposition, and graph Laplacian reduction

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

PBBFMM3D: A parallel black-box algorithm for kernel matrix-vector multiplication

Efficient Randomized Algorithms for the Fixed-Precision Low-Rank Matrix Approximation

A hybrid stochastic interpolation and compression method for kernel matrices

A general error analysis for randomized low-rank approximation with application to data assimilation

An Efficient Gauss--Newton Algorithm for Symmetric Low-Rank Product Matrix Approximations

Efficient kernel canonical correlation analysis using Nyström approximation

An Error-Resilient Redundant Subspace Correction Method

Improved Analyses of the Randomized Power Method and Block Lanczos Method