Abstract:The Kaczmarz algorithm is a simple iterative scheme for solving consistent linear systems. At each step, the method projects the current iterate onto the solution space of a single constraint. Hence, it requires low cost per iteration and storage, and it has a linear rate of convergence. Distributed implementations of Kaczmarz have recently become the de facto architectural choice for large-scale linear systems. Therefore, in this paper we develop a family of randomized block Kaczmarz algorithms that uses at each step a subset of the constraints and extrapolated stepsizes, and can be deployed on distributed computing units. Our approach is based on several new ideas and tools, including stochastic selection rules for the blocks of rows, stochastic conditioning of linear systems, and novel strategies for designing extrapolated stepsizes. We prove that randomized block Kaczmarz algorithms converge linearly in expectation, with a rate depending on the geometric properties of the matrix and its submatrices and on the size of the blocks. Our convergence analysis reveals that the algorithm is most effective when it is given a good sampling of the rows into well-conditioned blocks. Besides providing a general framework for the design and analysis of randomized block Kaczmarz methods, our results resolve an open problem in the literature related to the theoretical understanding of observed practical efficiency of extrapolated block Kaczmarz methods. We also propose an accelerated block Kaczmarz scheme, that is, acceleration in the sense of Chebyshev semi-iterative methods, where the stepsize is chosen based on the roots of Chebyshev polynomials, and we derive convergence rates depending on the square root of the geometric properties of the matrix. Finally, numerical examples illustrate the benefits of the new algorithms.

On constrained Kaczmarz algorithm with momentum for image reconstruction

On the adaptive deterministic block Kaczmarz method with momentum for solving large-scale consistent linear systems

Accelerated Kaczmarz Algorithms using History Information

Preasymptotic convergence of randomized Kaczmarz method

A greedy regularized block Kaczmarz method for accelerating reconstruction in magnetic particle imaging

Stochastic Momentum Method with Double Acceleration for Regularized Empirical Risk Minimization

Kaczmarz Iterative Projection and Nonuniform Sampling with Complexity Estimates

Convergence Analysis of Nonlinear Kaczmarz Method for Systems of Nonlinear Equations with Component-wise Convex Mapping

Nonlinear Kaczmarz algorithms and their convergence

The standard forms and convergence theory of the Kaczmarz-Tanabe type methods for solving linear systems

Multi-stage stochastic gradient method with momentum acceleration

Kaczmarz extended algorithm for tomographic image reconstruction from limited-data

Greedy randomized block Kaczmarz method for matrix equation AXB=C and its applications in color image restoration

A New Kaczmarz-Type Method And Its Acceleration For Nonlinear Ill-Posed Problems

An Accelerated Kaczmarz Type Method for Nonlinear Inverse Problems in Banach Spaces with Uniformly Convex Penalty

Randomized Kaczmarz with geometrically smoothed momentum

Faster Randomized Block Kaczmarz Algorithms

Iterative Hard Thresholding Based on Randomized Kaczmarz Method

Acceleration of Randomized Kaczmarz Method via the Johnson-Lindenstrauss Lemma

An Improved Reacceleration Optimization Algorithm Based on the Momentum Method for Image Recognition

Convergence Rates for Greedy Kaczmarz Algorithms, and Faster Randomized Kaczmarz Rules Using the Orthogonality Graph