Abstract:The goal of this paper is to study approaches to bridge the gap between first-order and second-order type methods for composite convex programs. Our key observations are: i) Many well-known operator splitting methods, such as forward-backward splitting (FBS) and Douglas-Rachford splitting (DRS), actually define a possibly semi-smooth and monotone fixed-point mapping; ii) The optimal solutions of the composite convex program and the solutions of the system of nonlinear equations derived from the fixed-point mapping are equivalent. Solving the system of nonlinear equations rediscovers a paradigm on developing second-order methods. Although these fixed-point mappings may not be differentiable, they are often semi-smooth and its generalized Jacobian matrix is positive semidefinite due to monotonicity. By combining a regularization approach and a known hyperplane projection technique, we propose an adaptive semi-smooth Newton method and establish its convergence to global optimality. A semi-smooth Levenberg-Marquardt (LM) method in terms of handling the nonlinear least squares formulation is further presented. In practice, the second-order methods can be activated until the first-order type methods reach a good neighborhood of the global optimal solution. Preliminary numerical results on Lasso regression, logistic regression, basis pursuit, linear programming and quadratic programming demonstrate that our second-order type algorithms are able to achieve quadratic or superlinear convergence as long as the fixed-point residual of the initial point is small enough.

Variational Properties of Decomposable Functions. Part II: Strong Second-Order Theory

Variational Properties of Decomposable Functions. Part I: Strict Epi-Calculus and Applications

Strong Variational Sufficiency for Nonlinear Semidefinite Programming and its Implications

Second Order Conditions to Decompose Smooth Functions as Sums of Squares

Strong variational sufficiency of nonsmooth optimization problems on Riemannian manifolds

Semi-Smooth Second-order Type Methods for Composite Convex Programs

On second-order variational analysis of variational convexity of prox-regular functions

Second-order optimality conditions for general nonconvex optimization problems and variational analysis of disjunctive systems

Exploring Jacobian Inexactness in Second-Order Methods for Variational Inequalities: Lower Bounds, Optimal Algorithms and Quasi-Newton Approximations

Variational Theory and Algorithms for a Class of Asymptotically Approachable Nonconvex Problems

A fresh look into variational analysis of $\mathcal C^2$-partly smooth functions

Generalized Metric Subregularity with Applications to High-Order Regularized Newton Methods

Convergence of the family of the deformed Euler–Halley iterations under the Hölder condition of the second derivative

Role of Subgradients in Variational Analysis of Polyhedral Functions

Variational Analysis of a Nonconvex and Nonsmooth Optimization Problem: An Introduction

New Generalized Derivatives for Solving Variational Inequalities Using the Nonsmooth Newton Methods

On the local convergence of the semismooth Newton method for composite optimization

Second-Order Strong Optimality and Second-Order Duality for Nonsmooth Constrained Multiobjective Fractional Programming Problems

A Regularized Semi-Smooth Newton Method with Projection Steps for Composite Convex Programs

Adaptive Methods for Variational Inequalities under Relaxed Smoothness Assumption

Second-order analysis of piecewise linear functions with applications to optimization and stability