Abstract:In this paper, we propose a quasi-Newton method for solving smooth and monotone nonlinear equations, including unconstrained minimization and minimax optimization as special cases. For the strongly monotone setting, we establish two global convergence bounds: (i) a linear convergence rate that matches the rate of the celebrated extragradient method, and (ii) an explicit global superlinear convergence rate that provably surpasses the linear convergence rate after at most ${O}(d)$ iterations, where $d$ is the problem's dimension. In addition, for the case where the operator is only monotone, we prove a global convergence rate of ${O}(\min\{{1}/{k},{\sqrt{d}}/{k^{1.25}}\})$ in terms of the duality gap. This matches the rate of the extragradient method when $k = {O}(d^2)$ and is faster when $k = \Omega(d^2)$. These results are the first global convergence results to demonstrate a provable advantage of a quasi-Newton method over the extragradient method, without querying the Jacobian of the operator. Unlike classical quasi-Newton methods, we achieve this by using the hybrid proximal extragradient framework and a novel online learning approach for updating the Jacobian approximation matrices. Specifically, guided by the convergence analysis, we formulate the Jacobian approximation update as an online convex optimization problem over non-symmetric matrices, relating the regret of the online problem to the convergence rate of our method. To facilitate efficient implementation, we further develop a tailored online learning algorithm based on an approximate separation oracle, which preserves structures such as symmetry and sparsity in the Jacobian matrices.

Extended Gauss-Newton and ADMM-Gauss-Newton Algorithms for Low-Rank Matrix Optimization

Gauss-Southwell type descent methods for low-rank matrix optimization

Gauss-Newton Methods for A Class of Nonsmooth Optimization Problems

An Alternating Direction Method with Continuation for Nonconvex Low Rank Minimization

An Efficient Gauss--Newton Algorithm for Symmetric Low-Rank Product Matrix Approximations

Generalized Nonconvex Nonsmooth Low-Rank Matrix Recovery Framework with Feasible Algorithm Designs and Convergence Analysis.

Accelerated Gradient Method for A Class of Nonconvex Low Rank Problem: Essentially Matching the Optimal Convex Convergence Rate

Nonconvex ADMM for Rank-Constrained Matrix Sensing Problem

Linearized ADMM for Nonconvex Nonsmooth Optimization with Convergence Analysis

A Multilevel Low-Rank Newton Method with Super-linear Convergence Rate and its Application to Non-convex Problems

Exact Gauss-Newton Optimization for Training Deep Neural Networks

ADMM for Nonsmooth Composite Optimization under Orthogonality Constraints

Online Learning Guided Quasi-Newton Methods with Global Non-Asymptotic Convergence

Provable accelerated gradient method for nonconvex low rank optimization

Alternating Direction and Taylor Expansion Minimization Algorithms for Unconstrained Nuclear Norm Optimization.

Good and Fast Row-Sparse ah-Symmetric Reflexive Generalized Inverses

A Unified Alternating Direction Method of Multipliers by Majorization Minimization

A Riemannian ADMM

On the (linear) convergence of Generalized Newton Inexact ADMM

Optimality Conditions and Numerical Algorithms for A Class of Linearly Constrained Minimax Optimization Problems

Stochastic linearized generalized alternating direction method of multipliers: Expected convergence rates and large deviation properties