Asymptotic convergence of restarted Anderson acceleration for certain normal linear systems

Hans De Sterck,Oliver A. Krzysik,Adam Smith

2024-07-04

Abstract:Anderson acceleration (AA) is widely used for accelerating the convergence of an underlying fixed-point iteration $\bm{x}_{k+1} = \bm{q}( \bm{x}_{k} )$, $k = 0, 1, \ldots$, with $\bm{x}_k \in \mathbb{R}^n$, $\bm{q} \colon \mathbb{R}^n \to \mathbb{R}^n$. Despite AA's widespread use, relatively little is understood theoretically about the extent to which it may accelerate the underlying fixed-point iteration. To this end, we analyze a restarted variant of AA with a restart size of one, a method closely related to GMRES(1). We consider the case of $\bm{q}( \bm{x} ) = M \bm{x} + \bm{b}$ with matrix $M \in \mathbb{R}^{n \times n}$ either symmetric or skew-symmetric. For both classes of $M$ we compute the worst-case root-average asymptotic convergence factor of the AA method, partially relying on conjecture in the symmetric setting, proving that it is strictly smaller than that of the underlying fixed-point iteration. For symmetric $M$, we show that the AA residual iteration corresponds to a fixed-point iteration for solving an eigenvector-dependent nonlinear eigenvalue problem (NEPv), and we show how this can result in the convergence factor strongly depending on the initial iterate, which we quantify exactly in certain special cases. Conversely, for skew-symmetric $M$ we show that the AA residual iteration is closely related to a power iteration for $M$, and how this results in the convergence factor being independent of the initial iterate. Supporting numerical results are given, which also indicate the theory is applicable to the more general setting of nonlinear $\bm{q}$ with Jacobian at the fixed point that is symmetric or skew symmetric.

Numerical Analysis

What problem does this paper attempt to address?

This paper attempts to address the problem of the asymptotic convergence behavior of the Anderson Acceleration (AA) method in specific linear systems. Specifically, the author focuses on the restarted Anderson acceleration method (restarted Anderson acceleration, rAA(1)), especially when the matrix \(M\in\mathbb{R}^{n\times n}\) in the iterative function \(q(x) = Mx + b\) is symmetric or skew - symmetric.

### Main research questions:
1. **Acceleration effect**: Can the Anderson acceleration method converge faster than the basic fixed - point iteration? If so, what is its convergence factor?
2. **Dependence on initial iteration**: Does the convergence factor of the Anderson acceleration method depend on the initial iteration value? If so, how can this dependence be quantified?
3. **Theoretical analysis and numerical verification**: Verify the convergence behavior of the Anderson acceleration method under different conditions through theoretical derivation and numerical experiments.

### Research background:
- **Fixed - point iteration**: For the equation \(x = q(x)\), the simplest iterative method is the fixed - point iteration \(x_{k + 1}=q(x_k)\).
- **Anderson acceleration**: To accelerate the convergence of the fixed - point iteration, one of the commonly used methods is Anderson acceleration, which adjusts the iteration step size by minimizing the residual.
- **Restarted Anderson acceleration**: This paper studies a restarted Anderson acceleration method. Each time it restarts, it only uses one previous iteration information (i.e., the restart size is 1), similar to the GMRES(1) method.

### Research methods:
1. **Symmetric matrix \(M\)**:
- Derived the residual propagation operator of the rAA(1) method and proved that its convergence factor is strictly less than that of the basic fixed - point iteration.
- Introduced the nonlinear eigenvalue problem depending on eigenvectors (NEPv) and showed that in some special cases, the residual vector of rAA(1) will exhibit a four - periodicity.

2. **Skew - symmetric matrix \(M\)**:
- Proved that the residual propagation of the rAA(1) method is closely related to the power iteration and that its convergence factor does not depend on the initial iteration value.

### Main conclusions:
- For a symmetric matrix \(M\), the worst - case root convergence factor of the rAA(1) method can be calculated through eigenvalues, and in some cases, its convergence speed is significantly better than that of the basic fixed - point iteration.
- For a skew - symmetric matrix \(M\), the convergence factor of the rAA(1) method is independent of the initial iteration value, and its convergence behavior is similar to that of the power iteration.

### Application prospects:
This research not only provides a theoretical basis for understanding the convergence characteristics of the Anderson acceleration method but also provides guidance for further optimization and application of this method. For example, in electronic structure calculations and other problems that require accelerating the solution of linear systems, the rAA(1) method may have important practical application value.

### Summary of mathematical formulas:
- Residual definition: \[r_k = x_k - q(x_k)\]
- Anderson acceleration formula: \[x_{k + 1}=q(x_k)+\sum_{i = 1}^{m_k}\beta_k^{(i)}(q(x_k)-q(x_{k - i}))\]
- Least - squares problem: \[\beta_k=\arg\min_{\beta\in\mathbb{R}^{m_k}}\left\|r_k+\sum_{i = 1}^{m_k}\beta^{(i)}(r_k - r_{k - i})\right\|^2\]
- Residual propagation operator: \[R(v)=M\left[I-\alpha(v)A\right]v\]
- Convergence factor: \[\varrho(r_0)=\limsup_{k\rightarrow\infty}\left(\|r_k\|^{1/k}\right)\]

Through these formulas and...