Abstract:Recent years have seen a growing interest in understanding acceleration methods through the lens of ordinary differential equations (ODEs). Despite the theoretical advancements, translating the rapid convergence observed in continuous-time models to discrete-time iterative methods poses significant challenges. In this paper, we present a comprehensive framework integrating the inertial systems with Hessian-driven damping equation (ISHD) and learning-based approaches for developing optimization methods through a deep synergy of theoretical insights. We first establish the convergence condition for ensuring the convergence of the solution trajectory of ISHD. Then, we show that provided the stability condition, another relaxed requirement on the coefficients of ISHD, the sequence generated through the explicit Euler discretization of ISHD converges, which gives a large family of practical optimization methods. In order to select the best optimization method in this family for certain problems, we introduce the stopping time, the time required for an optimization method derived from ISHD to achieve a predefined level of suboptimality. Then, we formulate a novel learning to optimize (L2O) problem aimed at minimizing the stopping time subject to the convergence and stability condition. To navigate this learning problem, we present an algorithm combining stochastic optimization and the penalty method (StoPM). The convergence of StoPM using the conservative gradient is proved. Empirical validation of our framework is conducted through extensive numerical experiments across a diverse set of optimization problems. These experiments showcase the superior performance of the learned optimization methods.

Safeguarded Learned Convex Optimization

Online Successive Convex Approximation for Two-Stage Stochastic Nonconvex Optimization

Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning

Learning to optimize with convergence guarantees using nonlinear system theory

ODE-based Learning to Optimize

Greedy Learning to Optimize with Convergence Guarantees

Learning to Optimize: A Primer and A Benchmark

Learning A Minimax Optimizer: A Pilot Study.

Learning for Robust Combinatorial Optimization: Algorithm and Application

Learning to optimize: A tutorial for continuous and mixed-integer optimization

Towards Constituting Mathematical Structures for Learning to Optimize

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

Distributed Learning with Convex SUM-of -Non-convex Objective

M-L2O: Towards Generalizable Learning-to-Optimize by Test-Time Fast Self-Adaptation

Accelerating Multi-Block Constrained Optimization Through Learning to Optimize

From Learning to Optimize to Learning Optimization Algorithms

Bayesian Optimization with Formal Safety Guarantees via Online Conformal Prediction

Learning-Augmented Decentralized Online Convex Optimization in Networks

Differentiable Linearized ADMM

Safeguarding adaptive methods: global convergence of Barzilai-Borwein and other stepsize choices