Abstract:Recent years have seen a growing interest in understanding acceleration methods through the lens of ordinary differential equations (ODEs). Despite the theoretical advancements, translating the rapid convergence observed in continuous-time models to discrete-time iterative methods poses significant challenges. In this paper, we present a comprehensive framework integrating the inertial systems with Hessian-driven damping equation (ISHD) and learning-based approaches for developing optimization methods through a deep synergy of theoretical insights. We first establish the convergence condition for ensuring the convergence of the solution trajectory of ISHD. Then, we show that provided the stability condition, another relaxed requirement on the coefficients of ISHD, the sequence generated through the explicit Euler discretization of ISHD converges, which gives a large family of practical optimization methods. In order to select the best optimization method in this family for certain problems, we introduce the stopping time, the time required for an optimization method derived from ISHD to achieve a predefined level of suboptimality. Then, we formulate a novel learning to optimize (L2O) problem aimed at minimizing the stopping time subject to the convergence and stability condition. To navigate this learning problem, we present an algorithm combining stochastic optimization and the penalty method (StoPM). The convergence of StoPM using the conservative gradient is proved. Empirical validation of our framework is conducted through extensive numerical experiments across a diverse set of optimization problems. These experiments showcase the superior performance of the learned optimization methods.

Learning to Sample Initial Solution for Solving 0–1 Discrete Optimization Problem by Local Search

Deep Learning for Explicitly Modeling Optimization Landscapes

Learning Multiple Initial Solutions to Optimization Problems

Data-efficient Supervised Learning is Powerful for Neural Combinatorial Optimization

Learning Sampling Policy for Faster Derivative Free Optimization

Teaching Networks to Solve Optimization Problems

Solving Large-Scale Multiobjective Optimization Problems With Sparse Optimal Solutions via Unsupervised Neural Networks

VLSs: A Local Search Algorithm for Distributed Constraint Optimization Problems

Gumbel-softmax-based Optimization: a Simple General Framework for Optimization Problems on Graphs

Score-Based Methods for Discrete Optimization in Deep Learning

A Deep Reinforcement Learning Approach to Efficient Distributed Optimization

A Hybrid Method of Combinatorial Search and Coordinate Descent for Discrete Optimization

Learning to Solve Combinatorial Optimization Problems on Real-World Graphs in Linear Time

Dynamic Strongly Convex Sparse Operator with Learning Mechanism for Sparse Large-Scale Multi-Objective Optimization

Tackling Prevalent Conditions in Unsupervised Combinatorial Optimization: Cardinality, Minimum, Covering, and More

Learning to Control Local Search for Combinatorial Optimization

Learning a Large Neighborhood Search Algorithm for Mixed Integer Programs

ODE-based Learning to Optimize

Learning to Balance Exploration and Exploitation in Pareto Local Search for Multi-Objective Combinatorial Optimization.

Adaptive Partitioning Strategy for High-Dimensional Discrete Simulation-based Optimization Problems

A Mathematics-Inspired Learning-to-Optimize Framework for Decentralized Optimization