Abstract:First-order methods are often analyzed via their continuous-time models, where their worst-case convergence properties are usually approached via Lyapunov functions. In this work, we provide a systematic and principled approach to find and verify Lyapunov functions for classes of ordinary and stochastic differential equations. More precisely, we extend the performance estimation framework, originally proposed by Drori and Teboulle [10], to continuous-time models. We retrieve convergence results comparable to those of discrete methods using fewer assumptions and convexity inequalities, and provide new results for stochastic accelerated gradient flows.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is about a systematic method for worst - case convergence analysis in optimization algorithms, especially for the search and verification of Lyapunov functions in continuous - time models. Specifically, the authors aim to provide a systematic and principled method to find and verify Lyapunov functions applicable to ordinary differential equation (ODE) and stochastic differential equation (SDE) categories. This helps to understand the behavior of optimization methods in continuous time and provides more concise and intuitive proofs for discrete - time methods. ### Main problems 1. **Worst - case convergence analysis**: How to systematically analyze the worst - case convergence of continuous - time models (such as gradient flows and their accelerated versions). 2. **Search and verification of Lyapunov functions**: How to systematically find and verify Lyapunov functions applicable to these models through techniques such as semidefinite programming (SDP). 3. **Expansion of performance estimation framework**: Expand the performance estimation framework proposed by Drori and Teboulle from discrete - time models to continuous - time models to reduce the use of assumptions and inequalities. ### Specific content - **Gradient flow**: The convergence properties of gradient flows and their accelerated versions are studied, verified by quadratic Lyapunov functions, and formulated as the feasibility problem of linear matrix inequalities (LMI). - **Stochastic optimization algorithm**: The trajectory properties of the stochastic gradient method approximated by SDE are analyzed, and the trade - off between initial condition forgetting and noise reduction is explored. - **Accelerated gradient flow**: For strongly convex functions, the linear convergence guarantees of the Polyak damped oscillator are derived, and these guarantees are verified by LMI. ### Methodological contributions - **Systematic Lyapunov analysis**: By introducing Gram matrices and semidefinite programming, a systematic method for finding and verifying Lyapunov functions is provided. - **Numerical tools**: Numerical tools are developed for selecting Lyapunov parameters to achieve the worst - case linear convergence rate. ### Practical applications This research not only provides a new perspective for the theoretical analysis of optimization algorithms, but also provides guidance for the design of optimization methods in practical applications. Especially when dealing with large - scale data sets, the performance of optimization algorithms can be evaluated and improved more effectively. ### Summary The main goal of this paper is to analyze the worst - case convergence in continuous - time models by using a systematic method, Lyapunov functions and semidefinite programming, and to provide more concise proofs for discrete - time methods. This method not only simplifies the analysis process, but also provides new tools and ideas for the design and improvement of optimization algorithms.

A systematic approach to Lyapunov analyses of continuous-time models in convex optimization

Automated tight Lyapunov analysis for first-order methods

A unified Euler--Lagrange system for analyzing continuous-time accelerated gradient methods

A fast continuous time approach for non-smooth convex optimization with time scaling and Tikhonov regularization

An Abstract Lyapunov Control Optimizer: Local Stabilization and Global Convergence

On the connections between optimization algorithms, Lyapunov functions, and differential equations: theory and insights

A fast continuous time approach with time scaling for nonsmooth convex optimization

Stochastic Differential Equations for Modeling First Order Optimization Methods

A new use of the Kurdyka-Lojasiewicz property to study asymptotic behaviours of some stochastic optimization algorithms in a non-convex differentiable framework

A Tutorial on a Lyapunov-Based Approach to the Analysis of Iterative Optimization Algorithms

Variational Analysis Perspective on Linear Convergence of Some First Order Methods for Nonsmooth Convex Optimization Problems

Automated Lyapunov Analysis of Primal-Dual Optimization Algorithms: An Interpolation Approach

Principled analyses and design of first-order methods with inexact proximal operators

A Continuous-Time Perspective on Global Acceleration for Monotone Equation Problems

Continuous Dynamics Related to Monotone Inclusions and Non-Smooth Optimization Problems

Frequency-Domain Representation of First-Order Methods: A Simple and Robust Framework of Analysis

Another Approach to Build Lyapunov Functions for the First Order Methods in the Quadratic Case

Continuous and discrete-time accelerated methods for an inequality constrained convex optimization problem

A convex optimization approach to the Lyapunov exponents

Finite-Time Nonconvex Optimization Using Time-Varying Dynamical Systems