Abstract:Support vector machines (SVMs) are successful supervised learning models that analyze data for classification and regression. Previous work has demonstrated the superiority of the SVMs in dealing with the high dimensional, low sample size problems. However, the increase of the sample size brings great challenges to accurately and efficiently solve the large-scale SVMs, especially for the nonlinear kernel SVMs, which may lead to huge computational costs and unaffordable storage burden. In this paper, we propose a highly efficient sparse semismooth Newton (SsN) based augmented Lagrangian (AL) method for solving a class of large-scale SVMs that can be formulated as a convex quadratic programming problem with a linear equality constraint and a simple box constraint. The asymptotic superlinear convergence rate of both the primal and the dual iteration sequences generated by the AL method is guaranteed due to the piecewise linear-quadratic structure of the problem. Furthermore, we reveal the close connection between the number of support vectors and the sparse structure of the generalized Jacobian for the inner subproblem of the AL method. By exploiting this hidden sparsity, the inner subproblem can be solved by the SsN method efficiently and accurately, which greatly reduces the storage burden and computational costs. In particular, for the nonlinear kernel SVMs, since the sparse structure may not manifest in the early iterations of the AL method, we solve a linear kernel SVM approximated by the random Fourier features method to produce a good initial point, and then transfer to solve the original problem. Numerical experiments demonstrate that the proposed algorithm outperforms the current state-of-the-art solvers for the large-scale SVMs.

A Dual Semismooth Newton Based Augmented Lagrangian Method for Large-Scale Linearly Constrained Sparse Group Square-Root Lasso Problems

An efficient Hessian based algorithm for solving large-scale sparse group Lasso problems

A sparse semismooth Newton based augmented Lagrangian method for large-scale support vector machines

Square-Root Lasso with Nonconvex Regularization: an ADMM Approach

An Efficient Algorithm for a Class of Large-Scale Support Vector Machines Exploiting Hidden Sparsity.

Efficient sparse Hessian based algorithms for the clustered lasso problem.

Tssnalm: A Fast Two-Stage Semi-Smooth Newton Augmented Lagrangian Method for Sparse CCA.

A Primal Majorized Semismooth Newton-CG Augmented Lagrangian Method for Large-Scale Linearly Constrained Convex Programming

A Newton-CG Augmented Lagrangian Method for Semidefinite Programming

Fast X-ray CT image reconstruction using the linearized augmented Lagrangian method with ordered subsets

An Augmented Lagrangian Primal-Dual Semismooth Newton Method for Multi-Block Composite Optimization

A Sparse-Group Lasso

A quadratically convergent semismooth Newton method for nonlinear semidefinite programming without generalized Jacobian regularity

Inexact Augmented Lagrangian Methods for Conic Programs: Quadratic Growth and Linear Convergence

Generalized Newton–Raphson Algorithm for High Dimensional LASSO Regression

A Proximal Point Algorithm For Log-Determinant Optimization With Group Lasso Regularization

Square Root LASSO: Well-posedness, Lipschitz stability and the tuning trade off

Group Sparse Recovery Via the $\ell ^0(\ell ^2)$ Penalty: Theory and Algorithm

Square Root LASSO: Well-Posedness, Lipschitz Stability, and the Tuning Trade-Off

An inexact augmented Lagrangian algorithm for unsymmetric saddle-point systems

Smoothing composite proximal gradient algorithm for sparse group Lasso problems with nonsmooth loss functions