Abstract:The goal of this paper is to study approaches to bridge the gap between first-order and second-order type methods for composite convex programs. Our key observations are: i) Many well-known operator splitting methods, such as forward-backward splitting (FBS) and Douglas-Rachford splitting (DRS), actually define a possibly semi-smooth and monotone fixed-point mapping; ii) The optimal solutions of the composite convex program and the solutions of the system of nonlinear equations derived from the fixed-point mapping are equivalent. Solving the system of nonlinear equations rediscovers a paradigm on developing second-order methods. Although these fixed-point mappings may not be differentiable, they are often semi-smooth and its generalized Jacobian matrix is positive semidefinite due to monotonicity. By combining a regularization approach and a known hyperplane projection technique, we propose an adaptive semi-smooth Newton method and establish its convergence to global optimality. A semi-smooth Levenberg-Marquardt (LM) method in terms of handling the nonlinear least squares formulation is further presented. In practice, the second-order methods can be activated until the first-order type methods reach a good neighborhood of the global optimal solution. Preliminary numerical results on Lasso regression, logistic regression, basis pursuit, linear programming and quadratic programming demonstrate that our second-order type algorithms are able to achieve quadratic or superlinear convergence as long as the fixed-point residual of the initial point is small enough.

On the local convergence of the semismooth Newton method for composite optimization

On the local convergence of the semismooth Newton method for composite optimization

On the Local Convergence of a Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization

Gauss-Newton Methods for A Class of Nonsmooth Optimization Problems

On Convergence of the Gauss-Newton Method for Convex Composite Optimization

A globally convergent proximal Newton-type method in nonsmooth convex optimization

Convergence analysis of the Gauss–Newton method for convex inclusion and convex-composite optimization problems

A projected semismooth Newton method for a class of nonconvex composite programs with strong prox-regularity

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

On Local Behavior of Newton-Type Methods Near Critical Solutions of Constrained Equations

Majorizing Functions and Convergence of the Gauss-Newton Method for Convex Composite Optimization

A trust region-type normal map-based semismooth Newton method for nonsmooth nonconvex composite optimization

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

Global non-asymptotic super-linear convergence rates of regularized proximal quasi-Newton methods on non-smooth composite problems

Local Convergence of Inexact Methods under the Hölder Condition

A Regularized Semi-Smooth Newton Method with Projection Steps for Composite Convex Programs

Gauss–Newton Method for Convex Composite Optimizations on Riemannian Manifolds

Convergence Behavior of Gauss-Newton's Method and Extensions of the Smale Point Estimate Theory.

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

Convergence Criterion of Newton's Method for Singular Systems with Constant Rank Derivatives

Semi-Smooth Second-order Type Methods for Composite Convex Programs