Abstract:Classification and probability estimation have broad applications in modern machine learning and data science applications, including biology, medicine, engineering, and computer science. The recent development of a class of weighted Support Vector Machines (wSVMs) has shown great values in robustly predicting the class probability and classification for various problems with high accuracy. The current framework is based on the $\ell^2$-norm regularized binary wSVMs optimization problem, which only works with dense features and has poor performance at sparse features with redundant noise in most real applications. The sparse learning process requires a prescreen of the important variables for each binary wSVMs for accurately estimating pairwise conditional probability. In this paper, we proposed novel wSVMs frameworks that incorporate automatic variable selection with accurate probability estimation for sparse learning problems. We developed efficient algorithms for effective variable selection for solving either the $\ell^1$-norm or elastic net regularized binary wSVMs optimization problems. The binary class probability is then estimated either by the $\ell^2$-norm regularized wSVMs framework with selected variables or by elastic net regularized wSVMs directly. The two-step approach of $\ell^1$-norm followed by $\ell^2$-norm wSVMs show a great advantage in both automatic variable selection and reliable probability estimators with the most efficient time. The elastic net regularized wSVMs offer the best performance in terms of variable selection and probability estimation with the additional advantage of variable grouping in the compensation of more computation time for high dimensional problems. The proposed wSVMs-based sparse learning methods have wide applications and can be further extended to $K$-class problems through ensemble learning.

High‐dimensional sparse classification using exponential weighting with empirical hinge loss

Robust Estimation and Shrinkage in Ultrahigh Dimensional Expectile Regression with Heavy Tails and Variance Heterogeneity

Expectile regression for analyzing heteroscedasticity in high dimension

Semiparametric Expectile Regression for High-dimensional Heavy-tailed and Heterogeneous Data

High-dimensional prediction for count response via sparse exponential weights

On high-dimensional classification by sparse generalized Bayesian logistic regression

High-dimensional classification by sparse logistic regression

High-dimensional Measurement Error Models for Lipschitz Loss

Weighted Lasso Estimates for Sparse Logistic Regression: Non-Asymptotic Properties with Measurement Errors

Sparse Multicategory Generalized Distance Weighted Discrimination in Ultra-High Dimensions

Inference for High-Dimensional Linear Expectile Regression with De-Biasing Method

High-Dimensional Sparse Additive Hazards Regression

A minimax optimal approach to high-dimensional double sparse linear regression

Sparse additive machine with pinball loss

High Dimensional Classification with combined Adaptive Sparse PLS and Logistic Regression

Sparse Learning and Class Probability Estimation with Weighted Support Vector Machines

Sparse Reduced Rank Huber Regression in High Dimensions

Class-Distributed Learning for Multinomial Logistic Regression with High Dimensional Features and a Large Number of Classes

Spike-and-Slab LASSO Generalized Additive Models and Scalable Algorithms for High-Dimensional Data Analysis

Robust adaptive LASSO in high-dimensional logistic regression

Sparse Exponential Discriminant Analysis