Abstract:We provide a complete characterization of the randomized Kaczmarz algorithm (RKA) for inconsistent linear systems. The Kaczmarz algorithm, known in some fields as the algebraic reconstruction technique, is a classical method for solving large-scale overdetermined linear systems through a sequence of projection operators; the randomized Kaczmarz algorithm is a recent proposal by Strohmer and Vershynin to randomize the sequence of projections in order to guarantee exponential convergence (in mean square) to the solutions. A flurry of work followed this development, with renewed interest in the algorithm, its extensions, and various bounds on their performance. Earlier, we studied the special case of consistent linear systems and provided an exact formula for the mean squared error (MSE) in the value reconstructed by RKA, as well as a simple way to compute the exact decay rate of the error. In this work, we consider the case of inconsistent linear systems, which is a more relevant scenario for most applications. First, by using a "lifting trick", we derive an exact formula for the MSE given a fixed noise vector added to the measurements. Then we show how to average over the noise when it is drawn from a distribution with known first and second-order statistics. Finally, we demonstrate the accuracy of our exact MSE formulas through numerical simulations, which also illustrate that previous upper bounds in the literature may be several orders of magnitude too high.

Refined upper bounds for the convergence of the randomized extended Kaczmarz and Gauss-Seidel algorithms

Convergence properties of the randomized extended Gauss-Seidel and Kaczmarz methods

On Multi-Step Randomized Extended Kaczmarz Method for Solving Large Sparse Inconsistent Linear Systems

Randomized Kaczmarz iteration methods: Algorithmic extensions and convergence theory

Randomized double and triple Kaczmarz for solving extended normal equations

On the Error Estimate of the Randomized Double Block Kaczmarz Method

On Convergence Rates of Kaczmarz-Type Methods with Different Selection Rules of Working Rows

Randomized Kaczmarz Algorithm for Inconsistent Linear Systems: An Exact MSE Analysis

On the convergence analysis of the greedy randomized Kaczmarz method

Rates of convergence of randomized Kaczmarz algorithms in Hilbert spaces

A simple linear convergence analysis of the reshuffling Kaczmarz method

Pseudoinverse-free randomized block iterative algorithms for consistent and inconsistent linear systems

Randomized Kaczmarz method with adaptive stepsizes for inconsistent linear systems

Inertial randomized Kaczmarz algorithms for solving coherent linear systems

On the Randomized Kaczmarz Algorithm

Three double multi-step randomized extended Kaczmarz methods for solving large sparse inconsistent linear systems

Remarks on Kaczmarz Algorithm for Solving Consistent and Inconsistent System of Linear Equations

Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm

A quantile-based block Kaczmarz algorithm for solving large consistent linear systems

On the exponential convergence of the Kaczmarz algorithm

Regularized randomized iterative algorithms for factorized linear systems