Abstract:Randomized Kaczmarz-type methods are appealing for large-scale linear systems arising from big data problems. One of the keys of randomized Kaczmarz-type methods is how to effectively select working rows from the coefficient matrix. To the best of our knowledge, most of the randomized Kaczmarz-type methods need to compute probabilities for choosing working rows. However, when the amount of data is huge, the computation of probabilities will be inaccurate due to many factors such as rounding errors and data distribution. Moreover, in some popular randomized Kaczmarz methods, we have to scan all the rows of the data matrix in advance, or to compute residual of the linear system in each step. Hence, we have to access all the rows of the data matrix, which are unfavorable for big data problems. To overcome these difficulties, we first introduce a semi-randomized Kaczmarz method in which there is no need to compute probabilities explicitly. However, we still have to access all the rows of the matrix for the computation of residuals. To improve the semi-randomized Kaczmarz method further, inspired by Chebyshev's (weak) law of large numbers, we apply the simple sampling strategy to the semi-randomized Kaczmarz method, and propose a semi-randomized Kaczmarz method with simple random sampling. In the new method, there is no need to calculate probabilities explicitly and it is free of computing residuals of the linear system, and is free of constructing index sets via scanning residuals. Indeed, we only need to compute some elements of residuals corresponding to simple sampling sets, and a small portion of rows of the matrix are utilized. Convergence results are established to show the rationality and feasibility of the two proposed methods. Numerical experiments demonstrate the superiority of the new methods over many state-of-the-art randomized Kaczmarz methods for large-scale linear systems.

A simple linear convergence analysis of the reshuffling Kaczmarz method

On the convergence analysis of the greedy randomized Kaczmarz method

The linear convergence of the greedy randomized Kaczmarz method is deterministic

A Kaczmarz Method with Simple Random Sampling for Solving Large Linear Systems

Fast Convergence of Random Reshuffling under Over-Parameterization and the Polyak-Łojasiewicz Condition

Randomized Kaczmarz Algorithm for Inconsistent Linear Systems: An Exact MSE Analysis

Nonlinear Kaczmarz algorithms and their convergence

Rates of convergence of randomized Kaczmarz algorithms in Hilbert spaces

Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm

Randomized Kaczmarz method with adaptive stepsizes for inconsistent linear systems

An Accelerated Randomized Kaczmarz Method Via Low-Rank Approximation

On Convergence Rates of Kaczmarz-Type Methods with Different Selection Rules of Working Rows

Convergence of Sign-based Random Reshuffling Algorithms for Nonconvex Optimization

A semi-randomized Kaczmarz method with simple random sampling for large-scale linear systems

Paved with good intentions: Analysis of a randomized block Kaczmarz method

Stochastic Extragradient with Random Reshuffling: Improved Convergence for Variational Inequalities

Convergence Rates for Greedy Kaczmarz Algorithms, and Faster Randomized Kaczmarz Rules Using the Orthogonality Graph

Randomized Kaczmarz iteration methods: Algorithmic extensions and convergence theory

A Semi‐Randomized and Augmented Kaczmarz Method With Simple Random Sampling for Large‐Scale Inconsistent Linear Systems

Random Reshuffling with Momentum for Nonconvex Problems: Iteration Complexity and Last Iterate Convergence

Randomized Kaczmarz Converges Along Small Singular Vectors