Abstract:The sparsity driven classification technologies have attracted much attention in recent years, due to their capability of providing more compressive representations and clear interpretation. Two most popular classification approaches are support vector machines (SVMs) and kernel logistic regression (KLR), each having its own advantages. The sparsification of SVM has been well studied, and many sparse versions of 2-norm SVM, such as 1-norm SVM (1-SVM), have been developed. But, the sparsification of KLR has been less studied. The existing sparsification of KLR is mainly based on L 1 norm and L 2 norm penalties, which leads to the sparse versions that yield solutions not so sparse as it should be. A very recent study on L 1/2 regularization theory in compressive sensing shows that L 1/2 sparse modeling can yield solutions more sparse than those of 1 norm and 2 norm, and, furthermore, the model can be efficiently solved by a simple iterative thresholding procedure. The objective function dealt with in L 1/2 regularization theory is, however, of square form, the gradient of which is linear in its variables (such an objective function is the so-called linear gradient function). In this paper, through extending the linear gradient function of L 1/2 regularization framework to the logistic function, we propose a novel sparse version of KLR, the 1/2 quasi-norm kernel logistic regression (1/2-KLR). The version integrates advantages of KLR and L 1/2 regularization, and defines an efficient implementation scheme of sparse KLR. We suggest a fast iterative thresholding algorithm for 1/2-KLR and prove its convergence. We provide a series of simulations to demonstrate that 1/2-KLR can often obtain more sparse solutions than the existing sparsity driven versions of KLR, at the same or better accuracy level. The conclusion is also true even in comparison with sparse SVMs (1-SVM and 2-SVM). We show an exclusive advantage of 1/2-KLR that the regularization parameter in the algorithm can be adaptively set whenever the sparsity (correspondingly, the number of support vectors) is given, which suggests a methodology of comparing sparsity promotion capability of different sparsity driven classifiers. As an illustration of benefits of 1/2-KLR, we give two applications of 1/2-KLR in semi-supervised learning, showing that 1/2-KLR can be successfully applied to the classification tasks in which only a few data are labeled.

A Sparse Logistic Regression Framework by Difference of Convex Functions Programming

Minimax sparse logistic regression for very high-dimensional feature selection.

On Regularized Sparse Logistic Regression

Optimal Feature Selection for Sparse Linear Discriminant Analysis and Its Applications in Gene Expression Data

Feature selection with multi-class logistic regression

Nonconvex Sparse Logistic Regression Via Proximal Gradient Descent.

Nonconvex Sparse Logistic Regression with Weakly Convex Regularization

Efficient Feature Selection via $\ell _{2, 0}$ℓ2, 0-norm Constrained Sparse Regression.

A Variance Minimization Criterion to Feature Selection Using Laplacian Regularization

Robust and Efficient Linear Discriminant Analysis With L2, 1-Norm for Feature Selection

Sparse Kernel Logistic Regression Based on L 1/2 Regularization

An Experiment on Feature Selection using Logistic Regression

Robust and Efficient Linear Discriminant Analysis With <italic>L</italic><sub>2,1</sub>-Norm for Feature Selection

A new regularization path for logistic regression via linearized Bregman

A sparse extreme learning machine framework by continuous optimization algorithms and its application in pattern recognition.

Piecewise Linear Regression via a Difference of Convex Functions

MLSLR: Multilabel Learning via Sparse Logistic Regression

Local Sparse Discriminative Feature Selection

Feature Screening Strategy for Non-Convex Sparse Logistic Regression with Log Sum Penalty

Feature selection for high‐dimensional regression via sparse LSSVR based on Lp‐norm