Abstract:Anderson acceleration (AA) is widely used for accelerating the convergence of an underlying fixed-point iteration $\bm{x}_{k+1} = \bm{q}( \bm{x}_{k} )$, $k = 0, 1, \ldots$, with $\bm{x}_k \in \mathbb{R}^n$, $\bm{q} \colon \mathbb{R}^n \to \mathbb{R}^n$. Despite AA's widespread use, relatively little is understood theoretically about the extent to which it may accelerate the underlying fixed-point iteration. To this end, we analyze a restarted variant of AA with a restart size of one, a method closely related to GMRES(1). We consider the case of $\bm{q}( \bm{x} ) = M \bm{x} + \bm{b}$ with matrix $M \in \mathbb{R}^{n \times n}$ either symmetric or skew-symmetric. For both classes of $M$ we compute the worst-case root-average asymptotic convergence factor of the AA method, partially relying on conjecture in the symmetric setting, proving that it is strictly smaller than that of the underlying fixed-point iteration. For symmetric $M$, we show that the AA residual iteration corresponds to a fixed-point iteration for solving an eigenvector-dependent nonlinear eigenvalue problem (NEPv), and we show how this can result in the convergence factor strongly depending on the initial iterate, which we quantify exactly in certain special cases. Conversely, for skew-symmetric $M$ we show that the AA residual iteration is closely related to a power iteration for $M$, and how this results in the convergence factor being independent of the initial iterate. Supporting numerical results are given, which also indicate the theory is applicable to the more general setting of nonlinear $\bm{q}$ with Jacobian at the fixed point that is symmetric or skew symmetric.

Descent Properties of an Anderson Accelerated Gradient Method with Restarting

Descent Properties of an Anderson Accelerated Gradient Method With Restarting

Accelerated Gradient Methods with Gradient Restart: Global Linear Convergence

Anderson Acceleration Without Restart: A Novel Method with $n$-Step Super Quadratic Convergence Rate

A Fast Anderson-Chebyshev Acceleration for Nonlinear Optimization.

Anderson Acceleration in Nonsmooth Problems: Local Convergence via Active Manifold Identification

Restarted Nonconvex Accelerated Gradient Descent: No More Polylogarithmic Factor in the O(<SUP>-7/4</SUP>) Complexity

Accelerated gradient descent: A guaranteed bound for a heuristic restart strategy

Restart of accelerated first order methods with linear convergence under a quadratic functional growth condition

Globally Convergent Type-I Anderson Acceleration for Nonsmooth Fixed-Point Iterations

Asymptotic convergence of restarted Anderson acceleration for certain normal linear systems

On The Unified Design Of Accelerated Gradient Descent

Convergence Analysis of Accelerated Stochastic Gradient Descent under the Growth Condition

Damped Anderson acceleration with restarts and monotonicity control for accelerating EM and EM-like algorithms

Restarted Nonconvex Accelerated Gradient Descent: No More Polylogarithmic Factor in the O(ε-7/4) Complexity.

Restarted Nonconvex Accelerated Gradient Descent: No More Polylogarithmic Factor in the $O(\epsilon^{-7/4})$ Complexity

Anderson Acceleration with Truncated Gram–Schmidt

Anderson acceleration for iteratively reweighted $\ell_1$ algorithm

Enhancing Stochastic Gradient Descent: A Unified Framework and Novel Acceleration Methods for Faster Convergence

An Anderson-Chebyshev Mixing Method for Nonlinear Optimization

Parameter-Free Accelerated Gradient Descent for Nonconvex Minimization