Abstract:The Kaczmarz method and its variants, which are types of stochastic gradient descent (SGD) methods, have been extensively studied for their simplicity and efficiency in solving linear systems. Random reshuffling (RR), also known as SGD without replacement, is typically faster in practice than traditional SGD method. Although some convergence analysis results for RR apply to the reshuffling Kaczmarz method, they do not comprehensively characterize its convergence. In this paper, we present a new convergence analysis of the reshuffling Kaczmarz method and demonstrate that it can converge linearly to the unique least-norm solution of the linear system. Furthermore, the convergence upper bound is tight and does not depend on the dimension of the coefficient matrix.

What problem does this paper attempt to address?

This paper attempts to solve the problem of linear convergence analysis of the Randomly - Rearranged Kaczmarz method (RRK method). Specifically, the existing convergence analysis results regarding the Random - Rearrangement (RR) method cannot comprehensively describe the convergence characteristics of the RRK method. Although some analysis results are applicable to the RRK method, they usually rely on strongly convex objective functions or are only applicable to the average case. To solve these problems, the author proposes a new convergence analysis method and proves that the RRK method can linearly converge to the unique minimum - norm solution of the linear system, and this convergence upper bound is tight and does not depend on the dimension of the coefficient matrix. Moreover, this method can utilize all samples in each iteration, thus showing better performance in practical applications. ### Main contributions: 1. **Proof of linear convergence**: It is proved for the first time that the RRK method can achieve linear convergence for any type of coefficient matrix (whether full - rank or rank - deficient). 2. **Tightness of the convergence upper bound**: It is proved that the convergence upper bound is tight, that is, there exists a linear system such that the upper bound takes the equal sign. 3. **No need for strong convex assumptions**: Different from some existing analyses, this method does not need to assume that the objective function is strongly convex, so it has a wider range of applications. 4. **Dynamic step - size selection**: The RRK method can adopt a dynamic step - size, which is more flexible and efficient than the traditional fixed - step - size method. ### Paper structure: - **Introduction**: It introduces the importance of solving linear equations and their application scenarios, and reviews the development history of the Kaczmarz method and its variants. - **Randomly - Rearranged Kaczmarz method**: It describes in detail the iteration process of the RRK method and its relationship with the Stochastic Gradient Descent (SGD) method. - **Linear convergence analysis**: By introducing auxiliary variables and iteration matrices, the linear convergence of the RRK method is proved. - **Conclusion**: It summarizes the main results and discusses future research directions, such as combining the Gearhart - Koshy acceleration technique or other momentum acceleration techniques to further improve the algorithm. Through these contributions, this paper provides a new perspective for understanding the convergence characteristics of the Randomly - Rearranged Kaczmarz method and lays a theoretical foundation for further optimizing and applying this method.

A simple linear convergence analysis of the reshuffling Kaczmarz method

On the convergence analysis of the greedy randomized Kaczmarz method

A Kaczmarz Method with Simple Random Sampling for Solving Large Linear Systems

Fast Convergence of Random Reshuffling under Over-Parameterization and the Polyak-Łojasiewicz Condition

Randomized Kaczmarz Algorithm for Inconsistent Linear Systems: An Exact MSE Analysis

Nonlinear Kaczmarz algorithms and their convergence

Rates of convergence of randomized Kaczmarz algorithms in Hilbert spaces

Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm

Randomized Kaczmarz method with adaptive stepsizes for inconsistent linear systems

On Convergence Rates of Kaczmarz-Type Methods with Different Selection Rules of Working Rows

Convergence of Sign-based Random Reshuffling Algorithms for Nonconvex Optimization

A semi-randomized Kaczmarz method with simple random sampling for large-scale linear systems

Paved with good intentions: Analysis of a randomized block Kaczmarz method

Stochastic Extragradient with Random Reshuffling: Improved Convergence for Variational Inequalities

Convergence Rates for Greedy Kaczmarz Algorithms, and Faster Randomized Kaczmarz Rules Using the Orthogonality Graph

Randomized Kaczmarz iteration methods: Algorithmic extensions and convergence theory

A Semi‐Randomized and Augmented Kaczmarz Method With Simple Random Sampling for Large‐Scale Inconsistent Linear Systems

Random Reshuffling with Momentum for Nonconvex Problems: Iteration Complexity and Last Iterate Convergence

Randomized Kaczmarz Converges Along Small Singular Vectors

An accelerated randomized Kaczmarz algorithm

Randomized block Kaczmarz method with projection for solving least squares