Abstract:We develop R2N, a modified quasi-Newton method for minimizing the sum of a $\mathcal{C}^1$ function $f$ and a lower semi-continuous prox-bounded $h$. Both $f$ and $h$ may be nonconvex. At each iteration, our method computes a step by minimizing the sum of a quadratic model of $f$, a model of $h$, and an adaptive quadratic regularization term. A step may be computed by a variant of the proximal-gradient method. An advantage of R2N over trust-region (TR) methods is that proximal operators do not involve an extra TR indicator. We also develop the variant R2DH, in which the model Hessian is diagonal, which allows us to compute a step without relying on a subproblem solver when $h$ is separable. R2DH can be used as standalone solver, but also as subproblem solver inside R2N. We describe non-monotone variants of both R2N and R2DH. Global convergence of a first-order stationarity measure to zero holds without relying on local Lipschitz continuity of $\nabla f$, while allowing model Hessians to grow unbounded, an assumption particularly relevant to quasi-Newton models. Under Lipschitz-continuity of $\nabla f$, we establish a tight worst-case complexity bound of $O(1 / \epsilon^{2/(1 - p)})$ to bring said measure below $\epsilon > 0$, where $0 \leq p < 1$ controls the growth of model Hessians. The latter must not diverge faster than $|\mathcal{S}_k|^p$, where $\mathcal{S}_k$ is the set of successful iterations up to iteration $k$. When $p = 1$, we establish the tight exponential complexity bound $O(\exp(c \epsilon^{-2}))$ where $c > 0$ is a constant. We describe our Julia implementation and report numerical experience on a basis-pursuit problem, image denoising, minimum-rank matrix completion, and a nonlinear support vector machine. In particular, the minimum-rank problem cannot be solved directly at this time by a TR approach as corresponding proximal operators are not known analytically.

SEP-QN: Scalable and Extensible Proximal Quasi-Newton Method for Dirty Statistical Models

A GLOBALLY AND SUPERLIEARLY CONVERGENT FEASIBLE SQP ALGORITHM FOR DEGENERATE CONSTRAINED OPTIMIZATION

A Scalable and Extensible Framework for Superposition-Structured Models

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Proximal Quasi-Newton for Computationally Intensive L1-regularized M-estimators

A Proximal Stochastic Quasi-Newton Algorithm with Dynamical Sampling and Stochastic Line Search

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

N ov 2 01 6 A Proximal Stochastic Quasi-Newton Algorithm

A nonmonotone proximal quasi-Newton method for multiobjective optimization

Accelerated Quasi-Newton Proximal Extragradient: Faster Rate for Smooth Convex Optimization

A Stochastic Quasi-Newton Method for Non-convex Optimization with Non-uniform Smoothness

Online Learning Guided Quasi-Newton Methods with Global Non-Asymptotic Convergence

A Stochastic Extra-Step Quasi-Newton Method for Nonsmooth Nonconvex Optimization

A Proximal Modified Quasi-Newton Method for Nonsmooth Regularized Optimization

Proximal methods for structured nonsmooth optimization over Riemannian submanifolds

On Fast Convergence of Proximal Algorithms for SQRT-Lasso Optimization: Don't Worry About Its Nonsmooth Loss Function

Revisiting Sub-sampled Newton Methods

A Proximal-Gradient Method for Constrained Optimization

A projected semismooth Newton method for a class of nonconvex composite programs with strong prox-regularity

Proximal Quasi-Newton Method for Composite Optimization over the Stiefel Manifold.

Accelerated Proximal Subsampled Newton Method