Abstract:We provide a complete characterization of the randomized Kaczmarz algorithm (RKA) for inconsistent linear systems. The Kaczmarz algorithm, known in some fields as the algebraic reconstruction technique, is a classical method for solving large-scale overdetermined linear systems through a sequence of projection operators; the randomized Kaczmarz algorithm is a recent proposal by Strohmer and Vershynin to randomize the sequence of projections in order to guarantee exponential convergence (in mean square) to the solutions. A flurry of work followed this development, with renewed interest in the algorithm, its extensions, and various bounds on their performance. Earlier, we studied the special case of consistent linear systems and provided an exact formula for the mean squared error (MSE) in the value reconstructed by RKA, as well as a simple way to compute the exact decay rate of the error. In this work, we consider the case of inconsistent linear systems, which is a more relevant scenario for most applications. First, by using a "lifting trick", we derive an exact formula for the MSE given a fixed noise vector added to the measurements. Then we show how to average over the noise when it is drawn from a distribution with known first and second-order statistics. Finally, we demonstrate the accuracy of our exact MSE formulas through numerical simulations, which also illustrate that previous upper bounds in the literature may be several orders of magnitude too high.

Randomized Kaczmarz with geometrically smoothed momentum

Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm

Randomized Kaczmarz method with adaptive stepsizes for inconsistent linear systems

Stochastic Momentum Method with Double Acceleration for Regularized Empirical Risk Minimization

Acceleration of Randomized Kaczmarz Method via the Johnson-Lindenstrauss Lemma

Randomized Kaczmarz Algorithm for Inconsistent Linear Systems: An Exact MSE Analysis

Randomized Sparse Block Kaczmarz as Randomized Dual Block-Coordinate Descent

On the fast convergence of minibatch heavy ball momentum

Random Scaling and Momentum for Non-smooth Non-convex Optimization

Randomized block Kaczmarz method with projection for solving least squares

On the Randomized Kaczmarz Algorithm

Randomized Extended Kaczmarz for Solving Least-Squares

Generalized Gearhart-Koshy acceleration for the Kaczmarz method

Randomized Kaczmarz Converges Along Small Singular Vectors

Paved with good intentions: Analysis of a randomized block Kaczmarz method

Randomized Kaczmarz with tail averaging

Accelerated Kaczmarz Algorithms using History Information

Faster randomized block sparse Kaczmarz by averaging

A Note on the Behavior of the Randomized Kaczmarz Algorithm of Strohmer and Vershynin

On constrained Kaczmarz algorithm with momentum for image reconstruction

On the adaptive deterministic block Kaczmarz method with momentum for solving large-scale consistent linear systems