Abstract:The Kaczmarz method is a way to iteratively solve a linear system of equations $Ax = b$. One interprets the solution $x$ as the point where hyperplanes intersect and then iteratively projects an approximate solution onto these hyperplanes to get better and better approximations. We note a somewhat related idea: one could take two random hyperplanes and project one into the orthogonal complement of the other. This leads to a sequence of linear systems $A^{(k)} x = b^{(k)}$ which is fast to compute, preserves the original solution and whose small singular values grow like $\sigma_{\ell}(A^{(k)}) \sim \exp(k/n^2) \cdot \sigma_{\ell}(A)$.
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is to improve the performance of the Kaczmarz method when solving the linear equation system \(Ax = b\), especially by using a new pre - processing technique (called Kaczmarz Kac Walk) to improve the condition number of matrix \(A\), thus accelerating the iterative solution process. Specifically, the paper focuses on the following points:
1. **Increasing small singular values**: The paper proposes a method to gradually increase the small singular values of matrix \(A\), making matrix \(A\) closer to an orthogonal matrix, and then improving the efficiency of solving the linear system.
2. **Keeping the Frobenius norm unchanged**: During the process of improving matrix \(A\), ensure that its Frobenius norm remains unchanged, that is, \(\sum_{i = 1}^n\sigma_i(A)^2\) is unchanged, where \(\sigma_i(A)\) is the \(i\)-th singular value of matrix \(A\).
3. **Randomly selecting rows for projection**: The paper introduces a method of randomly selecting two rows and projecting them onto the orthogonal complement space of one of the row vectors to gradually improve the properties of matrix \(A\).
4. **Theoretical analysis and numerical experiments**: The paper not only provides theoretical proofs but also verifies the effectiveness of this method through numerical experiments, especially its performance in dealing with large - scale and ill - conditioned linear systems.
### Specific problem description
Given a linear equation system \(Ax = b\), where \(A\in\mathbb{R}^{m\times n}\), \(m\geq n\), and assume that \(A\) is full - rank. The Kaczmarz method approximates the true solution \(x\) by successively projecting the current solution onto each hyperplane. However, the traditional Kaczmarz method may converge slowly in some cases, especially when the condition number of matrix \(A\) is large.
### Proposed solution
The paper proposes the Kaczmarz Kac Walk method. This method gradually improves the condition number of matrix \(A\) through random selection and projection operations on the rows of matrix \(A\). The specific steps are as follows:
- Randomly select two rows \(A_i\) and \(A_j\).
- Project \(A_j\) onto the orthogonal complement space of \(A_i\) and renormalize.
- Update the corresponding right - hand side term \(b_j\).
In this way, the small singular values of matrix \(A\) will gradually increase, and the large singular values will decrease, making matrix \(A\) closer to an orthogonal matrix, and then accelerating the convergence speed of the Kaczmarz method.
### Main results
The main theoretical result of the paper shows that for the matrix \(A\) processed by Kaczmarz Kac Walk, the growth rate of its small singular values can be approximated as:
\[ \sigma_\ell(A(k))\sim e^{k / n^2}\cdot\sigma_\ell(A) \]
This shows that as the number of iterations \(k\) increases, the condition number of matrix \(A\) will gradually improve, thus improving the efficiency of solving the linear system.
In addition, the paper also verifies the effectiveness of this method in dealing with large - scale and ill - conditioned linear systems through numerical experiments.