Abstract:We develop R2N, a modified quasi-Newton method for minimizing the sum of a $\mathcal{C}^1$ function $f$ and a lower semi-continuous prox-bounded $h$. Both $f$ and $h$ may be nonconvex. At each iteration, our method computes a step by minimizing the sum of a quadratic model of $f$, a model of $h$, and an adaptive quadratic regularization term. A step may be computed by a variant of the proximal-gradient method. An advantage of R2N over trust-region (TR) methods is that proximal operators do not involve an extra TR indicator. We also develop the variant R2DH, in which the model Hessian is diagonal, which allows us to compute a step without relying on a subproblem solver when $h$ is separable. R2DH can be used as standalone solver, but also as subproblem solver inside R2N. We describe non-monotone variants of both R2N and R2DH. Global convergence of a first-order stationarity measure to zero holds without relying on local Lipschitz continuity of $\nabla f$, while allowing model Hessians to grow unbounded, an assumption particularly relevant to quasi-Newton models. Under Lipschitz-continuity of $\nabla f$, we establish a tight worst-case complexity bound of $O(1 / \epsilon^{2/(1 - p)})$ to bring said measure below $\epsilon > 0$, where $0 \leq p < 1$ controls the growth of model Hessians. The latter must not diverge faster than $|\mathcal{S}_k|^p$, where $\mathcal{S}_k$ is the set of successful iterations up to iteration $k$. When $p = 1$, we establish the tight exponential complexity bound $O(\exp(c \epsilon^{-2}))$ where $c > 0$ is a constant. We describe our Julia implementation and report numerical experience on a basis-pursuit problem, image denoising, minimum-rank matrix completion, and a nonlinear support vector machine. In particular, the minimum-rank problem cannot be solved directly at this time by a TR approach as corresponding proximal operators are not known analytically.

N ov 2 01 6 A Proximal Stochastic Quasi-Newton Algorithm

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

A Proximal Stochastic Quasi-Newton Algorithm with Dynamical Sampling and Stochastic Line Search

An inexact regularized proximal Newton method for nonconvex and nonsmooth optimization

Proximal Dogleg Opportunistic Majorization for Nonconvex and Nonsmooth Optimization

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization.

A Stochastic Extra-Step Quasi-Newton Method for Nonsmooth Nonconvex Optimization

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

A Stochastic Quasi-Newton Method for Non-convex Optimization with Non-uniform Smoothness

A Proximal Modified Quasi-Newton Method for Nonsmooth Regularized Optimization

A Proximal-Gradient Method for Constrained Optimization

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

Smoothed Proximal Lagrangian Method for Nonlinear Constrained Programs

A globally convergent proximal Newton-type method in nonsmooth convex optimization

A Symplectic Discretization Based Proximal Point Algorithm for Convex Minimization

A nonmonotone accelerated proximal gradient method with variable stepsize strategy for nonsmooth and nonconvex minimization problems

A Semismooth Newton Stochastic Proximal Point Algorithm with Variance Reduction

Stochastic Newton Proximal Extragradient Method

On Fast Convergence of Proximal Algorithms for SQRT-Lasso Optimization: Don't Worry About Its Nonsmooth Loss Function

A nonmonotone proximal quasi-Newton method for multiobjective optimization