Abstract:We propose a new framework to design and analyze accelerated methods that solve general monotone equation (ME) problems $F(x)=0$. Traditional approaches include generalized steepest descent methods and inexact Newton-type methods. If $F$ is uniformly monotone and twice differentiable, these methods achieve local convergence rates while the latter methods are globally convergent thanks to line search and hyperplane projection. However, a global rate is unknown for these methods. The variational inequality methods can be applied to yield a global rate that is expressed in terms of $\|F(x)\|$ but these results are restricted to first-order methods and a Lipschitz continuous operator. It has not been clear how to obtain global acceleration using high-order Lipschitz continuity. This paper takes a continuous-time perspective where accelerated methods are viewed as the discretization of dynamical systems. Our contribution is to propose accelerated rescaled gradient systems and prove that they are equivalent to closed-loop control systems. Based on this connection, we establish the properties of solution trajectories. Moreover, we provide a unified algorithmic framework obtained from discretization of our system, which together with two approximation subroutines yields both existing high-order methods and new first-order methods. We prove that the $p^{th}$-order method achieves a global rate of $O(k^{-p/2})$ in terms of $\|F(x)\|$ if $F$ is $p^{th}$-order Lipschitz continuous and the first-order method achieves the same rate if $F$ is $p^{th}$-order strongly Lipschitz continuous. If $F$ is strongly monotone, the restarted versions achieve local convergence with order $p$ when $p \geq 2$. Our discrete-time analysis is largely motivated by the continuous-time analysis and demonstrates the fundamental role that rescaled gradients play in global acceleration for solving ME problems.

Unified Acceleration of High-Order Algorithms under General Hölder Continuity

Towards Unified Acceleration of High-Order Algorithms under H\"{o}lder Continuity and Uniform Convexity

A Continuous-Time Perspective on Global Acceleration for Monotone Equation Problems

A Universal Accelerated Primal–Dual Method for Convex Optimization Problems

A Unified Analysis for the Subgradient Methods Minimizing Composite Nonconvex, Nonsmooth and Non-Lipschitz Functions.

Robust Hybrid Zero-Order Optimization Algorithms with Acceleration via Averaging in Time

Geometry of First-Order Methods and Adaptive Acceleration.

Safeguarding adaptive methods: global convergence of Barzilai-Borwein and other stepsize choices

A Lyapunov Analysis of Accelerated PDHG Algorithms

Geometry of First-Order Methods and Adaptive Acceleration

Accelerated Gradient Algorithms with Adaptive Subspace Search for Instance-Faster Optimization

Accelerated first-order methods for convex optimization with locally Lipschitz continuous gradient

Unified Convergence Analysis for Adaptive Optimization with Moving Average Estimator

An inexact Bregman proximal point method and its acceleration version for unbalanced optimal transport

OPTAMI: Global Superlinear Convergence of High-order Methods

A new unified framework for designing convex optimization methods with prescribed theoretical convergence estimates: A numerical analysis approach

Accelerated Bregman Proximal Gradient Methods for Relatively Smooth Convex Optimization

Accelerated Over-Relaxation Heavy-Ball Methods with Provable Acceleration and Global Convergence

Fast convex optimization via closed-loop time scaling of gradient dynamics

Accelerated Gradient Methods Combining Tikhonov Regularization with Geometric Damping Driven by the Hessian

Higher-order tensor methods for minimizing difference of convex functions