Abstract:The smoothly clipped absolute deviation (SCAD) and the minimax concave penalty (MCP)-penalized regression models are two important and widely used nonconvex sparse learning tools that can handle variable selection and parameter estimation simultaneously and thus have potential applications in various fields, such as mining biological data in high-throughput biomedical studies. Theoretically, these two models enjoy the oracle property even in the high-dimensional settings, where the number of predictors p may be much larger than the number of observations n. However, numerically, it is quite challenging to develop fast and stable algorithms due to their nonconvexity and nonsmoothness. In this article, we develop a fast algorithm for SCAD- and MCP-penalized learning problems. First, we show that the global minimizers of both models are roots of the nonsmooth equations. Then, a semismooth Newton (SSN) algorithm is employed to solve the equations. We prove that the SSN algorithm converges locally and superlinearly to the Karush-Kuhn- Tucker (KKT) points. The computational complexity analysis shows that the cost of the SSN algorithm per iteration is O(np). Combined with the warm-start technique, the SSN algorithm can be very efficient and accurate. Simulation studies and a real data example suggest that our SSN algorithm, with comparable solution accuracy with the coordinate descent (CD) and the difference of convex (DC) proximal Newton algorithms, is more computationally efficient.

A dual based semismooth Newton-type algorithm for solving large-scale sparse Tikhonov regularization problems

A Proximal Iteratively Regularized Gauss–Newton Method for Nonlinear Inverse Problems

A Dual Semismooth Newton Based Augmented Lagrangian Method for Large-Scale Linearly Constrained Sparse Group Square-Root Lasso Problems

Extended Newton-type Method for Inverse Singular Value Problems with Multiple And/or Zero Singular Values

Projected Newton Method for noise constrained Tikhonov regularization

Newton-type Methods for Inverse Singular Value Problems with Multiple Singular Values

A Special Modified Tikhonov Regularization Matrix for Discrete Ill-Posed Problems

Generalized Tikhonov regularization method for large-scale linear inverse problems

A nested primal–dual iterated Tikhonov method for regularized convex optimization

Super-Universal Regularized Newton Method

Reduced-Space Iteratively Reweighted Second-Order Methods for Nonconvex Sparse Regularization

A quadratically convergent semismooth Newton method for nonlinear semidefinite programming without generalized Jacobian regularity

Multi-parameter Tikhonov regularization with the ` sparsity constraint: ` convergence

A Regularized Newton Method for Monotone Nonlinear Equations and Its Application

A preconditioned Krylov subspace method for linear inverse problems with general-form Tikhonov regularization

A globally convergent proximal Newton-type method in nonsmooth convex optimization

A Multilevel Low-Rank Newton Method with Super-linear Convergence Rate and its Application to Non-convex Problems

An efficient Hessian based algorithm for solving large-scale sparse group Lasso problems

A Joint Bidiagonalization Based Algorithm for Large Scale Linear Discrete Ill-posed Problems in General-Form Regularization

A Semismooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning

A sparse semismooth Newton based augmented Lagrangian method for large-scale support vector machines