Abstract:Obtaining the solution of constrained optimization problems as a function of parameters is very important in a multitude of applications, such as control and planning. Solving such parametric optimization problems in real time can present significant challenges, particularly when it is necessary to obtain highly accurate solutions or batches of solutions. To solve these challenges, we propose a learning-based iterative solver for constrained optimization which can obtain very fast and accurate solutions by customizing the solver to a specific parametric optimization problem. For a given set of parameters of the constrained optimization problem, we propose a first step with a neural network predictor that outputs primal-dual solutions of a reasonable degree of accuracy. This primal-dual solution is then improved to a very high degree of accuracy in a second step by a learned iterative solver in the form of a neural network. A novel loss function based on the Karush-Kuhn-Tucker conditions of optimality is introduced, enabling fully self-supervised training of both neural networks without the necessity of prior sampling of optimizer solutions. The evaluation of a variety of quadratic and nonlinear parametric test problems demonstrates that the predictor alone is already competitive with recent self-supervised schemes for approximating optimal solutions. The second step of our proposed learning-based iterative constrained optimizer achieves solutions with orders of magnitude better accuracy than other learning-based approaches, while being faster to evaluate than state-of-the-art solvers and natively allowing for GPU parallelization.

Accelerated Forward-Backward Optimization Using Deep Learning

Accelerated Optimization in Deep Learning with a Proportional-Integral-derivative Controller

Linear Convergence of Forward-Backward Accelerated Algorithms without Knowledge of the Modulus of Strong Convexity

Optimal Adaptive and Accelerated Stochastic Gradient Descent

Stagewise Accelerated Stochastic Gradient Methods for Nonconvex Optimization

NALA: a Nesterov accelerated look-ahead optimizer for deep learning

Accelerated Gradient-free Neural Network Training by Multi-convex Alternating Optimization

Smoothing accelerated algorithm for constrained nonsmooth convex optimization problems

Accelerated First-Order Optimization under Nonlinear Constraints

Breaking the Convergence Barrier: Optimization via Fixed-Time Convergent Flows

Accelerated First-Order Optimization Algorithms for Machine Learning.

AA-DLADMM: An Accelerated ADMM-based Framework for Training Deep Neural Networks

Self-Supervised Learning of Iterative Solvers for Constrained Optimization

Deterministic and Stochastic Accelerated Gradient Method for Convex Semi-Infinite Optimization

Fast convex optimization via closed-loop time scaling of gradient dynamics

MUSIC: Accelerated Convergence for Distributed Optimization With Inexact and Exact Methods

Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training

Greedy Learning to Optimize with Convergence Guarantees

SANIA: Polyak-type Optimization Framework Leads to Scale Invariant Stochastic Algorithms