Abstract:Recent years have seen a growing interest in understanding acceleration methods through the lens of ordinary differential equations (ODEs). Despite the theoretical advancements, translating the rapid convergence observed in continuous-time models to discrete-time iterative methods poses significant challenges. In this paper, we present a comprehensive framework integrating the inertial systems with Hessian-driven damping equation (ISHD) and learning-based approaches for developing optimization methods through a deep synergy of theoretical insights. We first establish the convergence condition for ensuring the convergence of the solution trajectory of ISHD. Then, we show that provided the stability condition, another relaxed requirement on the coefficients of ISHD, the sequence generated through the explicit Euler discretization of ISHD converges, which gives a large family of practical optimization methods. In order to select the best optimization method in this family for certain problems, we introduce the stopping time, the time required for an optimization method derived from ISHD to achieve a predefined level of suboptimality. Then, we formulate a novel learning to optimize (L2O) problem aimed at minimizing the stopping time subject to the convergence and stability condition. To navigate this learning problem, we present an algorithm combining stochastic optimization and the penalty method (StoPM). The convergence of StoPM using the conservative gradient is proved. Empirical validation of our framework is conducted through extensive numerical experiments across a diverse set of optimization problems. These experiments showcase the superior performance of the learned optimization methods.

Properties and practicability of convergence-guaranteed optimization methods derived from weak discrete gradients

A new unified framework for designing convex optimization methods with prescribed theoretical convergence estimates: A numerical analysis approach

A unified Euler--Lagrange system for analyzing continuous-time accelerated gradient methods

Convergence Error Analysis of Reflected Gradient Langevin Dynamics for Globally Optimizing Non-Convex Constrained Problems

Convergence error analysis of reflected gradient Langevin dynamics for non-convex constrained optimization

Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization

A gradient descent akin method for constrained optimization: algorithms and applications

ODE-based Learning to Optimize

A geometric integration approach to smooth optimisation: Foundations of the discrete gradient method

Generalized-Hukuhara-Gradient Efficient-Direction Method to Solve Optimization Problems with Interval-valued Functions and its Application in Least Squares Problems

Dissipative Gradient Descent Ascent Method: A Control Theory Inspired Algorithm for Min-max Optimization

On the equivalence between SOR-type methods for linear systems and discrete gradient methods for gradient systems

Computational Convergence Analysis of Distributed Gradient Tracking for Smooth Convex Optimization Using Dissipativity Theory

Convergence Results of a Nested Decentralized Gradient Method for Non-strongly Convex Problems

DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method

Stochastic Differential Equations for Modeling First Order Optimization Methods

Continuous-time stochastic gradient descent for optimizing over the stationary distribution of stochastic differential equations

Decentralized Implicit Differentiation

A Homogenization Approach for Gradient-Dominated Stochastic Optimization

Convergence in High Probability of Distributed Stochastic Gradient Descent Algorithms

The Anytime Convergence of Stochastic Gradient Descent with Momentum: From a Continuous-Time Perspective