Abstract:Recent years have seen a growing interest in understanding acceleration methods through the lens of ordinary differential equations (ODEs). Despite the theoretical advancements, translating the rapid convergence observed in continuous-time models to discrete-time iterative methods poses significant challenges. In this paper, we present a comprehensive framework integrating the inertial systems with Hessian-driven damping equation (ISHD) and learning-based approaches for developing optimization methods through a deep synergy of theoretical insights. We first establish the convergence condition for ensuring the convergence of the solution trajectory of ISHD. Then, we show that provided the stability condition, another relaxed requirement on the coefficients of ISHD, the sequence generated through the explicit Euler discretization of ISHD converges, which gives a large family of practical optimization methods. In order to select the best optimization method in this family for certain problems, we introduce the stopping time, the time required for an optimization method derived from ISHD to achieve a predefined level of suboptimality. Then, we formulate a novel learning to optimize (L2O) problem aimed at minimizing the stopping time subject to the convergence and stability condition. To navigate this learning problem, we present an algorithm combining stochastic optimization and the penalty method (StoPM). The convergence of StoPM using the conservative gradient is proved. Empirical validation of our framework is conducted through extensive numerical experiments across a diverse set of optimization problems. These experiments showcase the superior performance of the learned optimization methods.

An ODE-based hybrid method based on the nonmonotone technique

An Efficient Interior-Point Algorithm with New Non-Monotone Line Search Filter Method for Nonlinear Constrained Programming

The convergence of conjugate gradient method with nonmonotone line search

An improved Dai‐Liao‐style hybrid conjugate gradient‐based method for solving unconstrained nonconvex optimization and extension to constrained nonlinear monotone equations

A Parallel Method With Hybrid Algorithms For Mixed Integer Nonlinear Programming

The Hybridization of Branch and Bound with Metaheuristics for Nonconvex Multiobjective Optimization

A Hybrid Genetic Algorithm for Solving Nonlinear Programming Problems with Inequality Constraints

An Effective Hybrid Algorithm Based On Simplex Search And Differential Evolution For Global Optimization

Hybridizing Differential Evolution and Nelder-Mead Simplex Algorithm for Global Optimization

A Hybridized Vector Optimal Algorithm for Multiobjective Designs of Inverse Problems

A class of nonmonotone Armijo-type line search method for unconstrained optimization

A hybrid model-based optimal control method for nonlinear systems using simultaneous dynamic optimization strategies

A simulation-based approach for solving optimisation problems with ODE-type steady state constraints

A hybrid algorithm for nonlinear minimax problems

Optimization of electromagnetic problems by using a hybrid algorithm combining, the annealing evolution algorithm with the Hooke-Jeeves algorithm

Hybrid Optimization of Optimal Power Flow by Combining the Semidefinite Programming and Nonlinear Programming

A hybrid BFGS-Like method for monotone operator equations with applications

A Branch-and-Bound Based Method for Solving Monotone Optimization Problems

A convergent hybrid three-term conjugate gradient method with sufficient descent property for unconstrained optimization

A HYBRID GLOBAL NUMERICAL OPTIMIZATION WITH COMBINATION OF EXTREMAL OPTIMIZATION AND SEQUENTIAL QUADRATIC PROGRAMMING

ODE-based Learning to Optimize