Abstract:We develop R2N, a modified quasi-Newton method for minimizing the sum of a $\mathcal{C}^1$ function $f$ and a lower semi-continuous prox-bounded $h$. Both $f$ and $h$ may be nonconvex. At each iteration, our method computes a step by minimizing the sum of a quadratic model of $f$, a model of $h$, and an adaptive quadratic regularization term. A step may be computed by a variant of the proximal-gradient method. An advantage of R2N over trust-region (TR) methods is that proximal operators do not involve an extra TR indicator. We also develop the variant R2DH, in which the model Hessian is diagonal, which allows us to compute a step without relying on a subproblem solver when $h$ is separable. R2DH can be used as standalone solver, but also as subproblem solver inside R2N. We describe non-monotone variants of both R2N and R2DH. Global convergence of a first-order stationarity measure to zero holds without relying on local Lipschitz continuity of $\nabla f$, while allowing model Hessians to grow unbounded, an assumption particularly relevant to quasi-Newton models. Under Lipschitz-continuity of $\nabla f$, we establish a tight worst-case complexity bound of $O(1 / \epsilon^{2/(1 - p)})$ to bring said measure below $\epsilon > 0$, where $0 \leq p < 1$ controls the growth of model Hessians. The latter must not diverge faster than $|\mathcal{S}_k|^p$, where $\mathcal{S}_k$ is the set of successful iterations up to iteration $k$. When $p = 1$, we establish the tight exponential complexity bound $O(\exp(c \epsilon^{-2}))$ where $c > 0$ is a constant. We describe our Julia implementation and report numerical experience on a basis-pursuit problem, image denoising, minimum-rank matrix completion, and a nonlinear support vector machine. In particular, the minimum-rank problem cannot be solved directly at this time by a TR approach as corresponding proximal operators are not known analytically.

Adaptive Quadratically Regularized Newton Method for Riemannian Optimization

Gauss–Newton Method for Convex Composite Optimizations on Riemannian Manifolds

A Cubic Regularized Newton's Method over Riemannian Manifolds

Inexact Adaptive Cubic Regularization Algorithms on Riemannian Manifolds and Application

Riemannian Adaptive Regularized Newton Methods with Hölder Continuous Hessians

Super-Universal Regularized Newton Method

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

On Adaptive Cubic Regularized Newton's Methods for Convex Optimization via Random Sampling

Krylov Cubic Regularized Newton: A Subspace Second-Order Method with Dimension-Free Convergence Rate

Adaptively Accelerating Cubic Regularized Newton's Methods for Convex Optimization via Random Sampling

Generalized Newton Method with Positive Definite Regularization for Nonsmooth Optimization Problems with Nonisolated Solutions

Structured Quasi-Newton Methods for Optimization with Orthogonality Constraints.

Accelerating Adaptive Cubic Regularization of Newton's Method Via Random Sampling

A quadratically convergent semismooth Newton method for nonlinear semidefinite programming without generalized Jacobian regularity

A Regularized Newton Method for Monotone Nonlinear Equations and Its Application

A hybrid inexact regularized Newton and negative curvature method

An Adaptive Regularized Proximal Newton-Type Methods for Composite Optimization over the Stiefel Manifold

A restricted memory quasi-Newton bundle method for nonsmooth optimization on Riemannian manifolds

A Proximal Modified Quasi-Newton Method for Nonsmooth Regularized Optimization

Riemannian optimization on the symplectic Stiefel manifold using second-order information

An inexact regularized proximal Newton method for nonconvex and nonsmooth optimization