Abstract:Support vector machines (SVMs) are successful supervised learning models that analyze data for classification and regression. Previous work has demonstrated the superiority of the SVMs in dealing with the high dimensional, low sample size problems. However, the increase of the sample size brings great challenges to accurately and efficiently solve the large-scale SVMs, especially for the nonlinear kernel SVMs, which may lead to huge computational costs and unaffordable storage burden. In this paper, we propose a highly efficient sparse semismooth Newton (SsN) based augmented Lagrangian (AL) method for solving a class of large-scale SVMs that can be formulated as a convex quadratic programming problem with a linear equality constraint and a simple box constraint. The asymptotic superlinear convergence rate of both the primal and the dual iteration sequences generated by the AL method is guaranteed due to the piecewise linear-quadratic structure of the problem. Furthermore, we reveal the close connection between the number of support vectors and the sparse structure of the generalized Jacobian for the inner subproblem of the AL method. By exploiting this hidden sparsity, the inner subproblem can be solved by the SsN method efficiently and accurately, which greatly reduces the storage burden and computational costs. In particular, for the nonlinear kernel SVMs, since the sparse structure may not manifest in the early iterations of the AL method, we solve a linear kernel SVM approximated by the random Fourier features method to produce a good initial point, and then transfer to solve the original problem. Numerical experiments demonstrate that the proposed algorithm outperforms the current state-of-the-art solvers for the large-scale SVMs.

Sparse Random Projection for Χ2 Kernel Linearization: Algorithm and Applications to Image Classification

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

Efficient $\chi ^{2}$ Kernel Linearization Via Random Feature Maps.

2-Dimensional Kernel Discriminant Analysis Based on Image Sampling and Regrouping

Linear spatial pyramid matching using sparse coding for image classification

Similarity-Based Image Classification Via Kernelized Sparse Representation

Image Classification Based on KPCA and SVM with Randomized Hyper-parameter Optimization

Image classification using kernel collaborative representation with regularized least square

Hash-SVM: Scalable Kernel Machines for Large-Scale Visual Classification

Sparse Multiple Kernel Learning: Minimax Rates with Random Projection

Soft Kernel Convex Hull Support Vector Machine for Large Scale Noisy Datasets

A Novel Face Recognition Algorithm Via Weighted Kernel Sparse Representation.

An Efficient Algorithm for a Class of Large-Scale Support Vector Machines Exploiting Hidden Sparsity.

Self-Explanatory Sparse Representation For Image Classification

An efficient Kernel-based matrixized least squares support vector machine

Sparse Approximation to the Eigensubspace for Discrimination

A Large Dimensional Analysis of Least Squares Support Vector Machines

Random matrices meet machine learning: a large dimensional analysis of LS-SVM

Sparse Universum Quadratic Surface Support Vector Machine Models for Binary Classification

Kernel-Free Quadratic Surface Regression for Multi-Class Classification

Recursive Least Squares Projection Twin Support Vector Machines for Nonlinear Classification