Abstract:In this paper, we propose a quasi-Newton method for solving smooth and monotone nonlinear equations, including unconstrained minimization and minimax optimization as special cases. For the strongly monotone setting, we establish two global convergence bounds: (i) a linear convergence rate that matches the rate of the celebrated extragradient method, and (ii) an explicit global superlinear convergence rate that provably surpasses the linear convergence rate after at most ${O}(d)$ iterations, where $d$ is the problem's dimension. In addition, for the case where the operator is only monotone, we prove a global convergence rate of ${O}(\min\{{1}/{k},{\sqrt{d}}/{k^{1.25}}\})$ in terms of the duality gap. This matches the rate of the extragradient method when $k = {O}(d^2)$ and is faster when $k = \Omega(d^2)$. These results are the first global convergence results to demonstrate a provable advantage of a quasi-Newton method over the extragradient method, without querying the Jacobian of the operator. Unlike classical quasi-Newton methods, we achieve this by using the hybrid proximal extragradient framework and a novel online learning approach for updating the Jacobian approximation matrices. Specifically, guided by the convergence analysis, we formulate the Jacobian approximation update as an online convex optimization problem over non-symmetric matrices, relating the regret of the online problem to the convergence rate of our method. To facilitate efficient implementation, we further develop a tailored online learning algorithm based on an approximate separation oracle, which preserves structures such as symmetry and sparsity in the Jacobian matrices.

Generalized Levenberg-Marquardt Neural Nets for Minimization of Quasiconvex Scalar Functions

A Neural Network Model For General Minimax Problem

Neural Network Solution for General Nonlinear Optimization Problems

The Controlled Conjugate Gradient Type Trajectory-Following Neural Net for Minimization of Nonconvex Functions.

Unified Control Liapunov Function Based Design of Neural Networks That Aim at Global Minimization of Nonconvex Functions

A discrete-time neural network for optimization problems with hybrid constraints.

Neural Control of Discrete Weak Formulations: Galerkin, Least-Squares and Minimal-Residual Methods with Quasi-Optimal Weights

Design of Second Order Neural Networks As Dynamical Control Systems That Aim to Minimize Nonconvex Scalar Functions

N ov 2 01 6 A Proximal Stochastic Quasi-Newton Algorithm

Online Learning Guided Quasi-Newton Methods with Global Non-Asymptotic Convergence

Lp quasi-norm minimization: algorithm and applications

A Neural Network for Solving Inverse Quasi-Variational Inequalities

On the saddle point problem for non-convex optimization

Comparative Study of the CG and HBF ODEs Used in the Global Minimization of Nonconvex Functions

A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimiax Optimization

Gauss Newton method for solving variational problems of PDEs with neural network discretizaitons

A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimax Optimization

Solving for a Quadratic Programming with a Quadratic Constraint Based on a Neural Network Frame

Error estimates of residual minimization using neural networks for linear PDEs

Representation and Regression Problems in Neural Networks: Relaxation, Generalization, and Numerics