Survey of a Class of Iterative Row-Action Methods: The Kaczmarz Method

Inês A. Ferreira,Juan A. Acebrón,José Monteiro

2024-04-09

Abstract:The Kaczmarz algorithm is an iterative method that solves linear systems of equations. It stands out among iterative algorithms when dealing with large systems for two reasons. First, at each iteration, the Kaczmarz algorithm uses a single equation, resulting in minimal computational work per iteration. Second, solving the entire system may only require the use of a small subset of the equations. These characteristics have attracted significant attention to the Kaczmarz algorithm. Researchers have observed that randomly choosing equations can improve the convergence rate of the algorithm. This insight led to the development of the Randomized Kaczmarz algorithm and, subsequently, several other variations emerged. In this paper, we extensively analyze the native Kaczmarz algorithm and many of its variations using large-scale dense random systems as benchmarks. Through our investigation, we have verified that, for consistent systems, various row sampling schemes can outperform both the original and Randomized Kaczmarz method. Specifically, sampling without replacement and using quasirandom numbers are the fastest techniques. However, for inconsistent systems, the Conjugate Gradient method for Least-Squares problems overcomes all variations of the Kaczmarz method for these types of systems.

Numerical Analysis

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to efficiently find solutions through iterative row - action methods (especially the Kaczmarz algorithm and its variants) when dealing with large - scale systems of linear equations. Specifically, the paper focuses on the impact of different row - sampling strategies on the performance of the Kaczmarz algorithm when dealing with large - scale dense random systems. The paper mentions that for consistent systems, different row - sampling schemes can outperform the original Kaczmarz method and the randomized Kaczmarz method. In particular, sampling without replacement and using quasi - random numbers are the fastest techniques. However, for inconsistent systems, the Conjugate Gradient method outperforms all variants of the Kaczmarz method in solving the least - squares problem.

### Main contributions of the paper:
1. **Algorithm analysis and comparison**: The paper analyzes in detail the original Kaczmarz algorithm and its multiple variants, especially their performance in large - scale dense random systems.
2. **Row - sampling strategies**: It studies the impact of different row - sampling strategies (such as random sampling, sampling without replacement, using quasi - random numbers, etc.) on the convergence speed of the algorithm.
3. **Performance evaluation**: It verifies through experiments the performance differences of different row - sampling strategies in consistent and inconsistent systems.
4. **Theoretical support**: It provides a theoretical convergence rate analysis, explaining why certain sampling strategies can accelerate the convergence of the algorithm.

### Key technical points:
- **Kaczmarz algorithm**: An iterative method that uses only one equation per iteration, has a small amount of computation, and is suitable for large - scale systems.
- **Randomized Kaczmarz algorithm**: Accelerates convergence by randomly selecting equations.
- **Block Kaczmarz algorithm**: Uses multiple equations in one iteration to further increase the convergence speed.
- **Extended Kaczmarz algorithm**: Applicable to inconsistent systems, combines the randomized orthogonal projection algorithm and the randomized Kaczmarz algorithm to solve the least - squares problem.
- **Double - block Kaczmarz algorithm**: Combines the block Kaczmarz algorithm and the extended Kaczmarz algorithm, and uses row and column blocks simultaneously to further accelerate convergence.

### Application background:
- **Medical imaging**: For example, in computed tomography (CT), it is necessary to reconstruct images from a large amount of data. The Kaczmarz algorithm and its variants perform well in real - time processing of large amounts of data.
- **Solving large - scale systems of linear equations**: In scientific computing and engineering applications, dealing with large - scale systems of linear equations is a fundamental problem, and the Kaczmarz algorithm provides an efficient solution.

Through these analyses and experiments, the paper provides an important reference for understanding and optimizing the performance of the Kaczmarz algorithm and its variants in practical applications.