Abstract:The sparsity driven classification technologies have attracted much attention in recent years, due to their capability of providing more compressive representations and clear interpretation. Two most popular classification approaches are support vector machines (SVMs) and kernel logistic regression (KLR), each having its own advantages. The sparsification of SVM has been well studied, and many sparse versions of 2-norm SVM, such as 1-norm SVM (1-SVM), have been developed. But, the sparsification of KLR has been less studied. The existing sparsification of KLR is mainly based on L 1 norm and L 2 norm penalties, which leads to the sparse versions that yield solutions not so sparse as it should be. A very recent study on L 1/2 regularization theory in compressive sensing shows that L 1/2 sparse modeling can yield solutions more sparse than those of 1 norm and 2 norm, and, furthermore, the model can be efficiently solved by a simple iterative thresholding procedure. The objective function dealt with in L 1/2 regularization theory is, however, of square form, the gradient of which is linear in its variables (such an objective function is the so-called linear gradient function). In this paper, through extending the linear gradient function of L 1/2 regularization framework to the logistic function, we propose a novel sparse version of KLR, the 1/2 quasi-norm kernel logistic regression (1/2-KLR). The version integrates advantages of KLR and L 1/2 regularization, and defines an efficient implementation scheme of sparse KLR. We suggest a fast iterative thresholding algorithm for 1/2-KLR and prove its convergence. We provide a series of simulations to demonstrate that 1/2-KLR can often obtain more sparse solutions than the existing sparsity driven versions of KLR, at the same or better accuracy level. The conclusion is also true even in comparison with sparse SVMs (1-SVM and 2-SVM). We show an exclusive advantage of 1/2-KLR that the regularization parameter in the algorithm can be adaptively set whenever the sparsity (correspondingly, the number of support vectors) is given, which suggests a methodology of comparing sparsity promotion capability of different sparsity driven classifiers. As an illustration of benefits of 1/2-KLR, we give two applications of 1/2-KLR in semi-supervised learning, showing that 1/2-KLR can be successfully applied to the classification tasks in which only a few data are labeled.

Nonconvex Sparse Logistic Regression with Weakly Convex Regularization

Nonconvex Sparse Logistic Regression Via Proximal Gradient Descent.

On Regularized Sparse Logistic Regression

Robust Sparse Recovery Via Non-Convex Optimization

A Smoothing Proximal Gradient Algorithm for Nonsmooth Convex Regression with Cardinality Penalty.

Reduced-Space Iteratively Reweighted Second-Order Methods for Nonconvex Sparse Regularization

Nonconvex Sparse Representation With Slowly Vanishing Gradient Regularizers

A Non-convex Approach for Sparse Recovery with Convergence Guarantee

High-dimensional Inference Via Lipschitz Sparsity-Yielding Regularizers.

Avoiding strict saddle points of nonconvex regularized problems

Sparse Recovery Via Nonconvex Regularized M-estimators over ℓ_q-Balls

Sparse Kernel Logistic Regression Based on L 1/2 Regularization

Fast Learning with Nonconvex L1-2 Regularization.

A Comparison of Algorithms for Learning with Nonconvex Regularization

The Convergence Guarantees of a Non-Convex Approach for Sparse Recovery.

Slowly Varying Regression under Sparsity

Robust Sparse Recovery Via Weakly Convex Optimization in Impulsive Noise

A SUCCESSIVE CONVEX APPROXIMATION APPROACH FOR SPARSE SOLUTIONS OF CONVEX PROGRAMS

A Fast Convex Conjugated Algorithm for Sparse Recovery

An Extended Newton-type Algorithm for $\ell_2$-Regularized Sparse Logistic Regression and Its Efficiency for Classifying Large-scale Datasets

Nonconvex Sparse Regularization for Deep Neural Networks and Its Optimality