Abstract:The sparsity driven classification technologies have attracted much attention in recent years, due to their capability of providing more compressive representations and clear interpretation. Two most popular classification approaches are support vector machines (SVMs) and kernel logistic regression (KLR), each having its own advantages. The sparsification of SVM has been well studied, and many sparse versions of 2-norm SVM, such as 1-norm SVM (1-SVM), have been developed. But, the sparsification of KLR has been less studied. The existing sparsification of KLR is mainly based on L 1 norm and L 2 norm penalties, which leads to the sparse versions that yield solutions not so sparse as it should be. A very recent study on L 1/2 regularization theory in compressive sensing shows that L 1/2 sparse modeling can yield solutions more sparse than those of 1 norm and 2 norm, and, furthermore, the model can be efficiently solved by a simple iterative thresholding procedure. The objective function dealt with in L 1/2 regularization theory is, however, of square form, the gradient of which is linear in its variables (such an objective function is the so-called linear gradient function). In this paper, through extending the linear gradient function of L 1/2 regularization framework to the logistic function, we propose a novel sparse version of KLR, the 1/2 quasi-norm kernel logistic regression (1/2-KLR). The version integrates advantages of KLR and L 1/2 regularization, and defines an efficient implementation scheme of sparse KLR. We suggest a fast iterative thresholding algorithm for 1/2-KLR and prove its convergence. We provide a series of simulations to demonstrate that 1/2-KLR can often obtain more sparse solutions than the existing sparsity driven versions of KLR, at the same or better accuracy level. The conclusion is also true even in comparison with sparse SVMs (1-SVM and 2-SVM). We show an exclusive advantage of 1/2-KLR that the regularization parameter in the algorithm can be adaptively set whenever the sparsity (correspondingly, the number of support vectors) is given, which suggests a methodology of comparing sparsity promotion capability of different sparsity driven classifiers. As an illustration of benefits of 1/2-KLR, we give two applications of 1/2-KLR in semi-supervised learning, showing that 1/2-KLR can be successfully applied to the classification tasks in which only a few data are labeled.

Sequential Safe Feature Elimination Rule for L 1-Regularized Regression with Kullback–Leibler Divergence

A Safe Feature Elimination Rule for L1-Regularized Logistic Regression.

Safe Feature Identification Rule for Fused Lasso by An Extra Dual Variable

A Variance Minimization Criterion to Feature Selection Using Laplacian Regularization

Safe Feature Elimination for the LASSO and Sparse Supervised Learning Problems

A Reconstructed Feasible Solution-Based Safe Feature Elimination Rule for Expediting Multi-Task Lasso

Sparse Kernel Logistic Regression Based on L 1/2 Regularization

Safe Feature Elimination in Sparse Supervised Learning

A Safe Screening Rule for Sparse Logistic Regression

Robustly Train Normalizing Flows Via KL Divergence Regularization

Feature Screening Strategy for Non-Convex Sparse Logistic Regression with Log Sum Penalty

Adaptive Dual Graphs and Non-Convex Constraint Based Embedded Feature Selection

Robust and Efficient Linear Discriminant Analysis With L2, 1-Norm for Feature Selection

Simultaneous Safe Feature and Sample Elimination for Sparse Support Vector Regression.

Learning with Selected Features

Safe Screening for Logistic Regression with $\ell_0$-$\ell_2$ Regularization

L(2,1) Regularized Correntropy For Robust Feature Selection

Efficient Online Learning for Large-Scale Sparse Kernel Logistic Regression

Online Efficient Learning with Quantized KLMS and L<inf>1</inf> Regularization

Mind the duality gap: safer rules for the Lasso