Abstract:We propose a new framework to design and analyze accelerated methods that solve general monotone equation (ME) problems $F(x)=0$. Traditional approaches include generalized steepest descent methods and inexact Newton-type methods. If $F$ is uniformly monotone and twice differentiable, these methods achieve local convergence rates while the latter methods are globally convergent thanks to line search and hyperplane projection. However, a global rate is unknown for these methods. The variational inequality methods can be applied to yield a global rate that is expressed in terms of $\|F(x)\|$ but these results are restricted to first-order methods and a Lipschitz continuous operator. It has not been clear how to obtain global acceleration using high-order Lipschitz continuity. This paper takes a continuous-time perspective where accelerated methods are viewed as the discretization of dynamical systems. Our contribution is to propose accelerated rescaled gradient systems and prove that they are equivalent to closed-loop control systems. Based on this connection, we establish the properties of solution trajectories. Moreover, we provide a unified algorithmic framework obtained from discretization of our system, which together with two approximation subroutines yields both existing high-order methods and new first-order methods. We prove that the $p^{th}$-order method achieves a global rate of $O(k^{-p/2})$ in terms of $\|F(x)\|$ if $F$ is $p^{th}$-order Lipschitz continuous and the first-order method achieves the same rate if $F$ is $p^{th}$-order strongly Lipschitz continuous. If $F$ is strongly monotone, the restarted versions achieve local convergence with order $p$ when $p \geq 2$. Our discrete-time analysis is largely motivated by the continuous-time analysis and demonstrates the fundamental role that rescaled gradients play in global acceleration for solving ME problems.

Convergence Rate Analysis of Continuous- and Discrete-Time Smoothing Gradient Algorithms

A unified Euler--Lagrange system for analyzing continuous-time accelerated gradient methods

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Convergence Analysis of a Finite Difference Scheme for the Gradient Flow associated with the ROF Model

A convergence analysis of the perturbed compositional gradient flow: averaging principle and normal deviations

Analysis of Stochastic Gradient Descent in Continuous Time

Concavifiability and convergence: necessary and sufficient conditions for gradient descent analysis

Finite-time and Fixed-time Convergence in Continuous-time Optimization.

On the equivalence between SOR-type methods for linear systems and discrete gradient methods for gradient systems

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Local Conditions for Global Convergence of Gradient Flows and Proximal Point Sequences in Metric Spaces

Convergence Analysis of a Class of Nonsmooth Gradient Systems.

Breaking the Convergence Barrier: Optimization via Fixed-Time Convergent Flows

The Modified MSA, a Gradient Flow and Convergence

The Convergence of a Numerical Method for Total Variation Flow

Convergence Error Analysis of Reflected Gradient Langevin Dynamics for Globally Optimizing Non-Convex Constrained Problems

Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigms

Convergence Analysis of Mixed Finite Element Approximations to Shape Gradients in the Stokes Equation

Convergence and Error Analysis for the Scalar Auxiliary Variable (SAV) Schemes to Gradient Flows

Computational Convergence Analysis of Distributed Gradient Tracking for Smooth Convex Optimization Using Dissipativity Theory

A Continuous-Time Perspective on Global Acceleration for Monotone Equation Problems