Abstract:Support vector machines (SVMs) are successful supervised learning models that analyze data for classification and regression. Previous work has demonstrated the superiority of the SVMs in dealing with the high dimensional, low sample size problems. However, the increase of the sample size brings great challenges to accurately and efficiently solve the large-scale SVMs, especially for the nonlinear kernel SVMs, which may lead to huge computational costs and unaffordable storage burden. In this paper, we propose a highly efficient sparse semismooth Newton (SsN) based augmented Lagrangian (AL) method for solving a class of large-scale SVMs that can be formulated as a convex quadratic programming problem with a linear equality constraint and a simple box constraint. The asymptotic superlinear convergence rate of both the primal and the dual iteration sequences generated by the AL method is guaranteed due to the piecewise linear-quadratic structure of the problem. Furthermore, we reveal the close connection between the number of support vectors and the sparse structure of the generalized Jacobian for the inner subproblem of the AL method. By exploiting this hidden sparsity, the inner subproblem can be solved by the SsN method efficiently and accurately, which greatly reduces the storage burden and computational costs. In particular, for the nonlinear kernel SVMs, since the sparse structure may not manifest in the early iterations of the AL method, we solve a linear kernel SVM approximated by the random Fourier features method to produce a good initial point, and then transfer to solve the original problem. Numerical experiments demonstrate that the proposed algorithm outperforms the current state-of-the-art solvers for the large-scale SVMs.

Efficient Sparse Least Squares Support Vector Machines for Regression

Sparse Representation Based on Projection Method in Online Least Squares Support Vector Machines

Sparse Least Squares Support Vector Machine for Function Estimation

Modeling of Isomerization of C-8 Aromatics by Online Least Squares Support Vector Machine

Least Squares Support Vector Machine with Self-Organizing Multiple Kernel Learning and Sparsity.

L1-Norm Least Squares Support Vector Regression Via the Alternating Direction Method of Multipliers

M-estimator Based Support Vector Machine and Its Application

Recursive Training Algorithm for One-Class Support Vector Machine Based on Active Set Method

Modeling method of least squares support vector regression based on vector base learning

Sparse Least Absolute Deviation Support Vector Machine

Iterative robust least square support vector machine for spectral analysis

Improved fuzzy sparse multi-class least squares support vector machine

An Efficient Algorithm for a Class of Large-Scale Support Vector Machines Exploiting Hidden Sparsity.

A Fast Iterative Single Data Approach to Training Unconstrained Least Squares Support Vector Machines

An unbiased LSSVM model for classification and regression

Least Square Generalized Support Vector Machines for Regression

Sparse Least Squares Low Rank Kernel Machines

A Non-Biased Form of Least Squares Support Vector Classifier and Its Fast Online Learning

Selective Recursive LSSVR and its Applications in Process Modeling

Least Squares Support Vector Machine Algorithm

Online Kernel Learning with a Near Optimal Sparsity Bound