Abstract:In this paper, a sparse learning algorithm, probabilistic classification vector machines (PCVMs), is proposed. We analyze relevance vector machines (RVMs) for classification problems and observe that adopting the same prior for different classes may lead to unstable solutions. In order to tackle this problem, a signed and truncated Gaussian prior is adopted over every weight in PCVMs, where the sign of prior is determined by the class label, i.e., +1 or -1. The truncated Gaussian prior not only restricts the sign of weights but also leads to a sparse estimation of weight vectors, and thus controls the complexity of the model. In PCVMs, the kernel parameters can be optimized simultaneously within the training algorithm. The performance of PCVMs is extensively evaluated on four synthetic data sets and 13 benchmark data sets using three performance metrics, error rate (ERR), area under the curve of receiver operating characteristic (AUC), and root mean squared error (RMSE). We compare PCVMs with soft-margin support vector machines (SVM(Soft)), hard-margin support vector machines (SVM(Hard)), SVM with the kernel parameters optimized by PCVMs (SVM(PCVM)), relevance vector machines (RVMs), and some other baseline classifiers. Through five replications of twofold cross-validation F test, i.e., 5 x 2 cross-validation F test, over single data sets and Friedman test with the corresponding post-hoc test to compare these algorithms over multiple data sets, we notice that PCVMs outperform other algorithms, including SVM(Soft), SVM(Hard), RVM, and SVM(PCVM), on most of the data sets under the three metrics, especially under AUC. Our results also reveal that the performance of SVM(PCVM) is slightly better than SVM(Soft), implying that the parameter optimization algorithm in PCVMs is better than cross validation in terms of performance and computational complexity. In this paper, we also discuss the superiority of PCVMs' formulation using maximum a posteriori (MAP) analysis and margin analysis, which explain the empirical success of PCVMs.

PAC-Bayes Analysis for Twin Support Vector Machines

Weighted Support Vector Machine for Classification with Uneven Training Class Sizes

Twin SVM for conditional probability estimation in binary and multiclass classification

An Effective Multiclass Twin Hypersphere Support Vector Machine and Its Practical Engineering Applications

Uncertainty-aware twin support vector machines

Universal Consistency of Twin Support Vector Machines.

Weighted Twin Support Vector Machines with Local Information and Its Application.

Granular Ball Twin Support Vector Machine

Comprehensive Review On Twin Support Vector Machines

Weighted least squares twin support vector machine based on density peaks

Insensitive Stochastic Gradient Twin Support Vector Machine for Large Scale Problems

Multiview learning with twin parametric margin SVM

Improved Generalized-Pinball-Loss-Based Laplacian Twin Support Vector Machine for Data Classification

Robust Twin Parametric Margin Support Vector Machine for Multiclass Classification

Probabilistic Classification Vector Machines.

CTSVM: A robust twin support vector machine with correntropy-induced loss function for binary classification problems

Twin K -class support vector classification with pinball loss

Weighted Posterior Probability Output for Support Vector Machines

Kreĭn twin support vector machines for imbalanced data classification

Twin Support Vector Machines Based on Chaotic Mapping Dung Beetle Optimization Algorithm

Improving Generalization of Complex Models under Unbounded Loss Using PAC-Bayes Bounds