Abstract:We provide a complete characterization of the randomized Kaczmarz algorithm (RKA) for inconsistent linear systems. The Kaczmarz algorithm, known in some fields as the algebraic reconstruction technique, is a classical method for solving large-scale overdetermined linear systems through a sequence of projection operators; the randomized Kaczmarz algorithm is a recent proposal by Strohmer and Vershynin to randomize the sequence of projections in order to guarantee exponential convergence (in mean square) to the solutions. A flurry of work followed this development, with renewed interest in the algorithm, its extensions, and various bounds on their performance. Earlier, we studied the special case of consistent linear systems and provided an exact formula for the mean squared error (MSE) in the value reconstructed by RKA, as well as a simple way to compute the exact decay rate of the error. In this work, we consider the case of inconsistent linear systems, which is a more relevant scenario for most applications. First, by using a "lifting trick", we derive an exact formula for the MSE given a fixed noise vector added to the measurements. Then we show how to average over the noise when it is drawn from a distribution with known first and second-order statistics. Finally, we demonstrate the accuracy of our exact MSE formulas through numerical simulations, which also illustrate that previous upper bounds in the literature may be several orders of magnitude too high.

Almost Sure Convergence of the Kaczmarz Algorithm with Random Measurements

Preasymptotic convergence of randomized Kaczmarz method

Rates of convergence of randomized Kaczmarz algorithms in Hilbert spaces

Randomized Kaczmarz Algorithm for Inconsistent Linear Systems: An Exact MSE Analysis

Randomized Kaczmarz Converges Along Small Singular Vectors

On the exponential convergence of the Kaczmarz algorithm

Acceleration of Randomized Kaczmarz Method via the Johnson-Lindenstrauss Lemma

A Note on the Behavior of the Randomized Kaczmarz Algorithm of Strohmer and Vershynin

Randomized Extended Kaczmarz for Solving Least-Squares

Convergence properties of the randomized extended Gauss-Seidel and Kaczmarz methods

Kaczmarz Iterative Projection and Nonuniform Sampling with Complexity Estimates

Error estimates of Kaczmarz and randomized Kaczmarz methods

Convergence Rates for Greedy Kaczmarz Algorithms, and Faster Randomized Kaczmarz Rules Using the Orthogonality Graph

On Convergence Rates of Kaczmarz-Type Methods with Different Selection Rules of Working Rows

Convergence rates for Kaczmarz-type algorithms

On the Error Estimate of the Randomized Double Block Kaczmarz Method

Randomized Kaczmarz method with adaptive stepsizes for inconsistent linear systems

Inertial randomized Kaczmarz algorithms for solving coherent linear systems

The Dual Kaczmarz Algorithm

Convergence Analysis on a Fast Iterative Phase Retrieval Algorithm Without Independence Assumption

Extension of Sparse Randomized Kaczmarz Algorithm for Multiple Measurement Vectors