Abstract:We develop R2N, a modified quasi-Newton method for minimizing the sum of a $\mathcal{C}^1$ function $f$ and a lower semi-continuous prox-bounded $h$. Both $f$ and $h$ may be nonconvex. At each iteration, our method computes a step by minimizing the sum of a quadratic model of $f$, a model of $h$, and an adaptive quadratic regularization term. A step may be computed by a variant of the proximal-gradient method. An advantage of R2N over trust-region (TR) methods is that proximal operators do not involve an extra TR indicator. We also develop the variant R2DH, in which the model Hessian is diagonal, which allows us to compute a step without relying on a subproblem solver when $h$ is separable. R2DH can be used as standalone solver, but also as subproblem solver inside R2N. We describe non-monotone variants of both R2N and R2DH. Global convergence of a first-order stationarity measure to zero holds without relying on local Lipschitz continuity of $\nabla f$, while allowing model Hessians to grow unbounded, an assumption particularly relevant to quasi-Newton models. Under Lipschitz-continuity of $\nabla f$, we establish a tight worst-case complexity bound of $O(1 / \epsilon^{2/(1 - p)})$ to bring said measure below $\epsilon > 0$, where $0 \leq p < 1$ controls the growth of model Hessians. The latter must not diverge faster than $|\mathcal{S}_k|^p$, where $\mathcal{S}_k$ is the set of successful iterations up to iteration $k$. When $p = 1$, we establish the tight exponential complexity bound $O(\exp(c \epsilon^{-2}))$ where $c > 0$ is a constant. We describe our Julia implementation and report numerical experience on a basis-pursuit problem, image denoising, minimum-rank matrix completion, and a nonlinear support vector machine. In particular, the minimum-rank problem cannot be solved directly at this time by a TR approach as corresponding proximal operators are not known analytically.

On a non-archimedean broyden method

Explicit Superlinear Convergence of Broyden's Method in Nonlinear Equations

Explicit Superlinear Convergence Rates of Broyden's Methods in Nonlinear Equations

Block Broyden's Methods for Solving Nonlinear Equations.

Greedy and Random Broyden's Methods with Explicit Superlinear Convergence Rates in Nonlinear Equations

A two-step Broyden-like method for nonlinear equations

Online Learning Guided Quasi-Newton Methods with Global Non-Asymptotic Convergence

A Proximal Modified Quasi-Newton Method for Nonsmooth Regularized Optimization

A Matrix-Free Newton Method

A Two-Point Newton Method suitable for non-convergent Cases and with Super-Quadratic Convergence

Inexact quasi-Newton methods under a relaxed metric regularity assumption

Convergence Criterion of Newton's Method for Singular Systems with Constant Rank Derivatives

Convergence of Newton'S Method for Systems of Equations with Constant Rank Derivatives

Super-Universal Regularized Newton Method

The Riemann hypothesis and dynamics of Backtracking New Q-Newton's method

Newton's Method for Sections on Riemannian Manifolds: Generalized Covariant Α-Theory

On Local Behavior of Newton-Type Methods Near Critical Solutions of Constrained Equations

Asymptotically Newton-Type Methods without Inverses for Solving Equations

On Two Iteration Methods for the Quadratic Matrix Equations

A Nonmonotone Broyden Method for Unconstrained Optimization

Faster Explicit Superlinear Convergence for Greedy and Random Quasi-Newton Methods