Abstract:We propose a new framework to design and analyze accelerated methods that solve general monotone equation (ME) problems $F(x)=0$. Traditional approaches include generalized steepest descent methods and inexact Newton-type methods. If $F$ is uniformly monotone and twice differentiable, these methods achieve local convergence rates while the latter methods are globally convergent thanks to line search and hyperplane projection. However, a global rate is unknown for these methods. The variational inequality methods can be applied to yield a global rate that is expressed in terms of $\|F(x)\|$ but these results are restricted to first-order methods and a Lipschitz continuous operator. It has not been clear how to obtain global acceleration using high-order Lipschitz continuity. This paper takes a continuous-time perspective where accelerated methods are viewed as the discretization of dynamical systems. Our contribution is to propose accelerated rescaled gradient systems and prove that they are equivalent to closed-loop control systems. Based on this connection, we establish the properties of solution trajectories. Moreover, we provide a unified algorithmic framework obtained from discretization of our system, which together with two approximation subroutines yields both existing high-order methods and new first-order methods. We prove that the $p^{th}$-order method achieves a global rate of $O(k^{-p/2})$ in terms of $\|F(x)\|$ if $F$ is $p^{th}$-order Lipschitz continuous and the first-order method achieves the same rate if $F$ is $p^{th}$-order strongly Lipschitz continuous. If $F$ is strongly monotone, the restarted versions achieve local convergence with order $p$ when $p \geq 2$. Our discrete-time analysis is largely motivated by the continuous-time analysis and demonstrates the fundamental role that rescaled gradients play in global acceleration for solving ME problems.

On the equivalence between SOR-type methods for linear systems and discrete gradient methods for gradient systems

A unified Euler--Lagrange system for analyzing continuous-time accelerated gradient methods

Geometric Interpretation of Several Classical Iterative Methods for Linear System of Equations and Diverse Relaxation Parameter of the Sor Method

Convergence Rate Analysis of Continuous- and Discrete-Time Smoothing Gradient Algorithms

A geometric integration approach to smooth optimisation: Foundations of the discrete gradient method

Linear gradient structures and discrete gradient methods for conservative/dissipative differential-algebraic equations

Stochastic Differential Equations for Modeling First Order Optimization Methods

An SOR Implicit Iterative Algorithm for Coupled Lyapunov Equations.

Higher order discrete gradient method by the discontinuous Galerkin time-stepping method

Compact Discrete Gradient Schemes for Nonlinear Schrödinger Equations

Discrete gradient structure of a second-order variable-step method for nonlinear integro-differential models

Properties and practicability of convergence-guaranteed optimization methods derived from weak discrete gradients

An $O(s^r)$-Resolution ODE Framework for Understanding Discrete-Time Algorithms and Applications to the Linear Convergence of Minimax Problems

Higher order numerical methods for SDEs without globally monotone coefficients

On the connections between optimization algorithms, Lyapunov functions, and differential equations: theory and insights

An SOR implicit iterative algorithm for coupled Riccati matrix equations in It stochastic systems with Markovian jump

A Continuous-Time Perspective on Global Acceleration for Monotone Equation Problems

The scalar auxiliary variable (SAV) approach for gradient flows

Computing subgradients of convex relaxations for solutions of parametric ordinary differential equations

A Functional Approach to Interpreting the Role of the Adjoint Equation in Machine Learning

A gradient discretisation method to analyse numerical schemes for non-linear variational inequalities, application to the seepage problem