Abstract:The sparsity driven classification technologies have attracted much attention in recent years, due to their capability of providing more compressive representations and clear interpretation. Two most popular classification approaches are support vector machines (SVMs) and kernel logistic regression (KLR), each having its own advantages. The sparsification of SVM has been well studied, and many sparse versions of 2-norm SVM, such as 1-norm SVM (1-SVM), have been developed. But, the sparsification of KLR has been less studied. The existing sparsification of KLR is mainly based on L 1 norm and L 2 norm penalties, which leads to the sparse versions that yield solutions not so sparse as it should be. A very recent study on L 1/2 regularization theory in compressive sensing shows that L 1/2 sparse modeling can yield solutions more sparse than those of 1 norm and 2 norm, and, furthermore, the model can be efficiently solved by a simple iterative thresholding procedure. The objective function dealt with in L 1/2 regularization theory is, however, of square form, the gradient of which is linear in its variables (such an objective function is the so-called linear gradient function). In this paper, through extending the linear gradient function of L 1/2 regularization framework to the logistic function, we propose a novel sparse version of KLR, the 1/2 quasi-norm kernel logistic regression (1/2-KLR). The version integrates advantages of KLR and L 1/2 regularization, and defines an efficient implementation scheme of sparse KLR. We suggest a fast iterative thresholding algorithm for 1/2-KLR and prove its convergence. We provide a series of simulations to demonstrate that 1/2-KLR can often obtain more sparse solutions than the existing sparsity driven versions of KLR, at the same or better accuracy level. The conclusion is also true even in comparison with sparse SVMs (1-SVM and 2-SVM). We show an exclusive advantage of 1/2-KLR that the regularization parameter in the algorithm can be adaptively set whenever the sparsity (correspondingly, the number of support vectors) is given, which suggests a methodology of comparing sparsity promotion capability of different sparsity driven classifiers. As an illustration of benefits of 1/2-KLR, we give two applications of 1/2-KLR in semi-supervised learning, showing that 1/2-KLR can be successfully applied to the classification tasks in which only a few data are labeled.

Kernel Logistic Regression Algorithm for Large-Scale Data Classification

Distance-Based Classifier Via the Kernel Trick.

Efficient Online Learning for Large-Scale Sparse Kernel Logistic Regression

Online sparse IR background estimation via KRLS

An Efficient Kernel-Based Nonlinear Regression Method For Two-Class Classification

Low-rank kernel regression with preserved locality for multi-class analysis

Kernel Generalized Nonlinear Discriminant Analysis Algorithm For Pattern Recognition

Sparse Kernel Logistic Regression Based on L 1/2 Regularization

A Novel Kernel-Based Maximum a Posteriori Classification Method

Scalable Kernel Logistic Regression with Nyström Approximation: Theoretical Analysis and Application to Discrete Choice Modelling

Selective Recursive Kernel Learning for Online Identification of Nonlinear Systems with NARX Form

Efficient Kernel Discriminant Analysis via Spectral Regression

Kernelized dual regression incorporating local information for image set classification

Gauss-like Logarithmic Kernel Function to improve the performance of kernel machines on the small datasets

Classification With Truncated Distance Kernel.

Large Scale Online Kernel Classification

Linear Regression-Based Efficient SVM Learning for Large-Scale Classification.

Kernel Based Simple Regularized Multiple Criteria Linear Programs for Binary Classification

Iterative Kernel Regression with Preconditioning

A Fast Kernel-Based Nonlinear Discriminant Analysis for Multi-Class Problems.

Speed up kernel discriminant analysis