Abstract:We propose a new framework to design and analyze accelerated methods that solve general monotone equation (ME) problems $F(x)=0$. Traditional approaches include generalized steepest descent methods and inexact Newton-type methods. If $F$ is uniformly monotone and twice differentiable, these methods achieve local convergence rates while the latter methods are globally convergent thanks to line search and hyperplane projection. However, a global rate is unknown for these methods. The variational inequality methods can be applied to yield a global rate that is expressed in terms of $\|F(x)\|$ but these results are restricted to first-order methods and a Lipschitz continuous operator. It has not been clear how to obtain global acceleration using high-order Lipschitz continuity. This paper takes a continuous-time perspective where accelerated methods are viewed as the discretization of dynamical systems. Our contribution is to propose accelerated rescaled gradient systems and prove that they are equivalent to closed-loop control systems. Based on this connection, we establish the properties of solution trajectories. Moreover, we provide a unified algorithmic framework obtained from discretization of our system, which together with two approximation subroutines yields both existing high-order methods and new first-order methods. We prove that the $p^{th}$-order method achieves a global rate of $O(k^{-p/2})$ in terms of $\|F(x)\|$ if $F$ is $p^{th}$-order Lipschitz continuous and the first-order method achieves the same rate if $F$ is $p^{th}$-order strongly Lipschitz continuous. If $F$ is strongly monotone, the restarted versions achieve local convergence with order $p$ when $p \geq 2$. Our discrete-time analysis is largely motivated by the continuous-time analysis and demonstrates the fundamental role that rescaled gradients play in global acceleration for solving ME problems.

A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights

Generalized Continuous-Time Models for Nesterov's Accelerated Gradient Methods

Accelerated optimization algorithms and ordinary differential equations: the convex non Euclidean case

Direct Runge-Kutta Discretization Achieves Acceleration.

A unified Euler--Lagrange system for analyzing continuous-time accelerated gradient methods

A More Stable Accelerated Gradient Method Inspired by Continuous-Time Perspective

Acceleration via Symplectic Discretization of High-Resolution Differential Equations

Differential Equations for Modeling Asynchronous Algorithms

Analysis Accelerated Mirror Descent via High-resolution ODEs

On adapting Nesterov's scheme to accelerate iterative methods for linear problems

Comparative analysis of accelerated gradient algorithms for convex optimization: high and super resolution ODE approach

Accelerating Nesterov's Method for Strongly Convex Functions with Lipschitz Gradient

Convergence Analysis of a Class of Nonsmooth Gradient Systems.

A Note on Nesterov's Accelerated Method in Nonconvex Optimization: a Weak Estimate Sequence Approach

Efficient, Accurate and Stable Gradients for Neural ODEs

A Continuous-Time Perspective on Global Acceleration for Monotone Equation Problems

Asymptotic error analysis for stochastic gradient optimization schemes with first and second order modified equations

Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigms

On The Unified Design Of Accelerated Gradient Descent

The Global R-linear Convergence of Nesterov's Accelerated Gradient Method with Unknown Strongly Convex Parameter

An SDE Perspective on Stochastic Inertial Gradient Dynamics with Time-Dependent Viscosity and Geometric Damping