Convergence rates for Kaczmarz-type algorithms

Constantin Popa

DOI: https://doi.org/10.48550/arXiv.1701.08002

2017-01-27

Abstract:In this paper we make a theoretical analysis of the convergence rates of Kaczmarz and Extended Kaczmarz projection algorithms for some of the most practically used control sequences. We first prove an at least linear convergence rate for the Kaczmarz-Tanabe and its Extended version methods (the one in which a complete set of projections using row/column index is performed in each iteration). Then we apply the main ideas of this analysis in establishing an at least sublinear, respectively linear convergence rate for the Kaczmarz algorithm with almost cyclic and the remotest set control strategies, and their extended versions, respectively. These results complete the existing ones related to the random selection procedures.

Numerical Analysis

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to analyze and determine the convergence rates of Kaczmarz - type algorithms and their extended versions under different control sequences. Specifically, the author has carried out a detailed theoretical analysis of the following types of algorithms:

1. **Kaczmarz - Tanabe (KT) algorithm**: This is a complete projection set algorithm, which uses the row index of each equation to perform one projection in each iteration.
2. **Extended Kaczmarz - Tanabe (EKT) algorithm**: This is an extended version of the KT algorithm, which is suitable for inconsistent least - squares problems.
3. **Single - projection Kaczmarz algorithm**: It includes different projection index selection strategies such as Almost Cyclic Kaczmarz (ACK), Maximal Residual Kaczmarz (MRK) and Random Kaczmarz (RK).
4. **Extended single - projection Kaczmarz algorithm**: Such as Extended Random Kaczmarz algorithm (REK), Extended Maximal Residual Kaczmarz algorithm (MREK) and Extended Almost Cyclic Kaczmarz algorithm (ACEK).

### Main contributions of the paper
- **Linear convergence rate**: It is proved that the KT, EKT, MRK and MREK algorithms have at least a linear convergence rate.
- **Sub - linear convergence rate**: It is proved that the ACK and ACEK algorithms have at least a sub - linear convergence rate.
- **Filling the gap**: These results supplement the existing literature on the linear convergence rate of random selection and the farthest - set control strategy.

### Formula summary
To ensure the correctness and readability of the formulas, the following are some key formulas involved in the paper:

- **Projection formula**:
\[
P_{H_i}(x) = x - \frac{\langle x, A_i \rangle - b_i}{\|A_i\|^2} A_i
\]

- **Definition of linear convergence**:
\[
\|x_k - \xi\| \leq \epsilon_k, \quad \forall k \geq 0
\]
where,
\[
\lim_{k \to \infty} \frac{\epsilon_{k+1}}{\epsilon_k} = \mu, \quad 0 < \mu < 1
\]

- **Update formula of KT algorithm**:
\[
x_{k+1} = (P_{H_1} \circ \cdots \circ P_{H_m})(x_k)
\]

- **Update formula of EKT algorithm**:
\[
y_{k+1} = \Phi y_k, \quad b_{k+1} = \hat{b} - y_{k+1}, \quad x_{k+1} = Q x_k + R b_{k+1}
\]

Through these analyses, the author provides a comprehensive understanding of the convergence properties of Kaczmarz - type algorithms and lays a foundation for further research.