Abstract:The goal of this paper is to study approaches to bridge the gap between first-order and second-order type methods for composite convex programs. Our key observations are: i) Many well-known operator splitting methods, such as forward-backward splitting (FBS) and Douglas-Rachford splitting (DRS), actually define a possibly semi-smooth and monotone fixed-point mapping; ii) The optimal solutions of the composite convex program and the solutions of the system of nonlinear equations derived from the fixed-point mapping are equivalent. Solving the system of nonlinear equations rediscovers a paradigm on developing second-order methods. Although these fixed-point mappings may not be differentiable, they are often semi-smooth and its generalized Jacobian matrix is positive semidefinite due to monotonicity. By combining a regularization approach and a known hyperplane projection technique, we propose an adaptive semi-smooth Newton method and establish its convergence to global optimality. A semi-smooth Levenberg-Marquardt (LM) method in terms of handling the nonlinear least squares formulation is further presented. In practice, the second-order methods can be activated until the first-order type methods reach a good neighborhood of the global optimal solution. Preliminary numerical results on Lasso regression, logistic regression, basis pursuit, linear programming and quadratic programming demonstrate that our second-order type algorithms are able to achieve quadratic or superlinear convergence as long as the fixed-point residual of the initial point is small enough.

The Second-Order Differential Equation System with the Feedback Controls for Solving Convex Programming

A second-order differential equation method for equilibrium programming with constraints

The Second-Order Differential Equation System with the Controlled Process for Variational Inequality with Constraints.

Two Differential Equation Algorithms for Solving Constrained Optimization Problems

Solving Nonconvex Optimal Control Problems by Convex Optimization

Second order Nonlinear Impulsive Integro-differential Equations of Mixed Type and Optimal Controls

A DIFFERENTIAL EQUATION METHOD FOR SOLVING NONLINEARLY CONSTRAINED OPTIMIZATION PROBLEMS

THE SECOND-ORDER DIFFERENTIAL EQUATION METHOD FOR SOLVING THE VARIATIONAL INEQUALITY PROBLEM

Linear-quadratic Optimal Control of Second-Order Parabolic Systems

Semi-Smooth Second-order Type Methods for Composite Convex Programs

The Equivalence Conditions of Optimal Feedback Control-Strategy Operators for Zero-Sum Linear Quadratic Stochastic Differential Game with Random Coefficients

Second-order Nonlinear Impulsive Integro-Differential Equations of Mixed Type with Time-Varying Generating Operators and Optimal Controls on Banach Spaces

Two Differential Equation Systems for Inequality Constrained Optimization

Second-order nonlocal impulsive integro-differential equations of mixed type and optimal controls

A perturbation approach for an inverse linear second-order cone programming

Second-order Necessary Condition of Optimal Control Problem Governed by ODE

Second-Order Nonlinear Impulsive Evolution Equations with Time-Varying Generating Operators and Optimal Controls

A Second Order Primal–Dual Dynamical System for a Convex–Concave Bilinear Saddle Point Problem

A successive convex optimization method for bilinear matrix inequality problems and its application to static output‐feedback control

A Differential System for Nonlinear Programming Problems

Second Order Nonlinear Impulsive Time-Variant Systems with Unbounded Perturbation and Optimal Controls