Abstract:This paper studies the commonly utilized windowed Anderson acceleration (AA) algorithm for fixed-point methods, $x^{(k+1)}=q(x^{(k)})$. It provides the first proof that when the operator $q$ is linear and symmetric the windowed AA, which uses a sliding window of prior iterates, improves the root-linear convergence factor over the fixed-point iterations. When $q$ is nonlinear, yet has a symmetric Jacobian at a fixed point, a slightly modified AA algorithm is proved to have an analogous root-linear convergence factor improvement over fixed-point iterations. Simulations verify our observations. Furthermore, experiments with different data models demonstrate AA is significantly superior to the standard fixed-point methods for Tyler's M-estimation.
What problem does this paper attempt to address?
### What problem does this paper attempt to solve?
This paper aims to study and prove the convergence performance of the Windowed Anderson Acceleration (AA) algorithm in symmetric fixed - point iterations. Specifically, the author focuses on whether the AA algorithm can improve the root - linear convergence factor of fixed - point iterations. The following are the main contributions of the paper:
1. **Case of linear symmetric operators**:
- When the operator \(q\) is linear and symmetric, the paper first proves that windowed AA can improve the root - linear convergence factor of fixed - point iterations by using a sliding window of previous iterations.
- Specifically, for the linear symmetric operator \(q(x)=Wx + a\), where \(W\in S^{n\times n}\) (i.e., \(W\) is a symmetric matrix) and \(\|W\|<1\), the paper gives an upper bound of the root - linear convergence factor of AA(m), indicating that its convergence speed is faster than that of the standard fixed - point iteration.
2. **Case of nonlinear operators**:
- When the operator \(q\) is nonlinear but has a symmetric Jacobian matrix at the fixed point, the paper proves that a slightly modified AA algorithm can also improve the root - linear convergence factor of fixed - point iterations.
- This result is proved by approximating \(q(x)\) with a local first - order Taylor expansion, and under specific conditions, the local root - linear convergence factor of the modified AA algorithm is strictly less than the convergence factor of the standard fixed - point iteration.
3. **Numerical experiment verification**:
- The paper verifies the theoretical results through numerical experiments, including experiments on linear and nonlinear operators. In particular, the paper shows the superior performance of the AA algorithm in dealing with the Tyler M - estimation (TME) problem, especially on high - dimensional data sets.
### Key formulas
- **Root - linear convergence factor**: For the sequence \(\{x^{(k)}\}\), its root - linear convergence factor is defined as:
\[
\rho\{x^{(k)}\}=\limsup_{k\rightarrow\infty}\left\|x^*-x^{(k)}\right\|^{1/k}
\]
- **Upper bound of the convergence factor of linear symmetric operators** (Theorem 1):
\[
\rho\{\text{AA}(m)\}\leq\sqrt{w_0\|W\|}
\]
where,
\[
w_0=\sup_{a\geq0}\sin\left(\sin^{-1}\left(\frac{|\lambda_{\max}(W)-\lambda_{\min}(W)|a}{\sqrt{4+(2 - (\lambda_{\max}(W)+\lambda_{\min}(W)))^2a^2}}\right)+\left|\tan^{-1}(a)-\tan^{-1}\left(\frac{a}{1-\frac{\lambda_{\max}(W)+\lambda_{\min}(W)}{2}}\right)\right|\right)
\]
### Summary
This paper solves the problem of the convergence performance of the Anderson acceleration algorithm in symmetric fixed - point iterations, provides strict mathematical proofs and numerical experiment support, indicating that the AA algorithm can significantly improve the convergence speed in many cases, especially for linear and nonlinear symmetric operators. This provides an important theoretical basis and practical application guidance for accelerating fixed - point iteration methods.