Abstract:The truncated regular L-1-loss support vector machine can eliminate the excessive number of support vectors (SVs); thus, it has significant advantages in robustness and scalability. However, in this paper, we discover that the associated state-of-the-art solvers, such as difference convex algorithm and concave-convex procedure, not only have limited sparsity promoting property for general truncated losses especially the L-2-loss but also have poor scalability for large-scale problems. To circumvent these drawbacks, we present a general multistage scheme with explicit interpretation regarding SVs as well as outliers. In particular, we solve the general nonconvex truncated loss minimization through a sequence of associated convex subproblems, in which the outliers are removed in advance. The proposed algorithm can be regarded as a structural optimization attempt carefully considering sparsity imposed by the nonconvex truncated losses. We show that this general multistage algorithm offers sufficient sparsity especially for the truncated L-2-loss. To further improve the scalability, we propose a linear multistep algorithm by employing a single iteration of coordinate descent to monotonically decrease the objective function at each stage and a kernel algorithm by using the Karush-Kuhn-Tucker conditions to cheaply find most part of the outliers for the next stage. Comparison experiments demonstrate that our methods have superiority in sparsity as well as efficiency in scalability.

Sparse Optimization Problem with S-Difference Regularization

A Forward-Backward Bregman Splitting Scheme for Regularized Distributed Optimization Problems

Sparse Signal Recovery Via Minimax-Concave Penalty and ℓ 1 -Norm Loss Function.

Difference-of-Convex Algorithms for a Class of Sparse Group $\ell_0$ Regularized Optimization Problems

Improving Sparsity and Scalability in Regularized Nonconvex Truncated-Loss Learning Problems

Robust Sparse Recovery Via Non-Convex Optimization

Avoiding strict saddle points of nonconvex regularized problems

Structured Sparsity Optimization with Non-Convex Surrogates of L2,0-Norm: A Unified Algorithmic Framework

Sparse Recovery Via Nonconvex Regularized M-estimators over ℓ_q-Balls

A Truncated Approximate Difference Algorithm for Sparse Signal Recovery

A Smoothing Proximal Gradient Algorithm for Nonsmooth Convex Regression with Cardinality Penalty.

A Unified Primal Dual Active Set Algorithm for Nonconvex Sparse Recovery

Structured Sparsity Optimization with Non-Convex Surrogates of $\ell _{2,0}$ℓ2,0-Norm: A Unified Algorithmic Framework

Group Sparse Recovery Via the $\ell ^0(\ell ^2)$ Penalty: Theory and Algorithm

Neural Network for a Class of Sparse Optimization with L0-regularization.

A SUCCESSIVE CONVEX APPROXIMATION APPROACH FOR SPARSE SOLUTIONS OF CONVEX PROGRAMS

An Alternating Proximal Splitting Method With Global Convergence For Nonconvex Structured Sparsity Optimization

Fast Sparse Recovery Via Non-Convex Optimization

A Non-convex Approach for Sparse Recovery with Convergence Guarantee

Smoothing Proximal Gradient Method for General Structured Sparse Learning