On Regularized Sparse Logistic Regression

Mengyuan Zhang,Kai Liu

DOI: https://doi.org/10.48550/arXiv.2309.05925

2023-10-12

Abstract:Sparse logistic regression is for classification and feature selection simultaneously. Although many studies have been done to solve $\ell_1$-regularized logistic regression, there is no equivalently abundant work on solving sparse logistic regression with nonconvex regularization term. In this paper, we propose a unified framework to solve $\ell_1$-regularized logistic regression, which can be naturally extended to nonconvex regularization term, as long as certain requirement is satisfied. In addition, we also utilize a different line search criteria to guarantee monotone convergence for various regularization terms. Empirical experiments on binary classification tasks with real-world datasets demonstrate our proposed algorithms are capable of performing classification and feature selection effectively at a lower computational cost.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the limitations of existing Sparse Logistic Regression methods when dealing with non - convex regularization terms. Specifically, although a great deal of research has focused on solving the logistic regression problem with ℓ1 regularization, relatively little research has been done on sparse logistic regression with non - convex regularization terms. In addition, existing methods often require strong assumptions when dealing with non - convex regularization terms, and these methods usually transform non - convex problems into convex problems for solution, which limits their generality and effectiveness.

To overcome these problems, this paper proposes a new unified framework for solving sparse logistic regression problems with ℓ1 regularization and non - convex regularization terms. This framework can not only be naturally extended to non - convex regularization terms, but also ensure that the algorithm can achieve monotonic convergence under various regularization terms by improving the step - size setting and line - search criteria. In addition, experimental results show that this framework can effectively perform classification and feature selection at a lower computational cost.

### Main contributions:
1. **Unified framework**: Proposed a unified framework that can handle ℓ1 regularization and non - convex regularization terms simultaneously.
2. **Improved optimization algorithm**: Based on the Iterative Shrinkage - Thresholding Algorithm (ISTA) and its accelerated version FISTA, the step - size setting and line - search criteria are improved, making the algorithm applicable to convex and non - convex regularization terms.
3. **Support for non - convex regularization terms**: Extended the application range of non - convex penalty terms and compared the performance of different regularization terms.
4. **Experimental verification**: Proved the effectiveness and efficiency of the proposed algorithm in binary classification tasks through experiments, especially its performance on high - dimensional data sets.

### Formula summary:
- Objective function:
\[
\min_{\beta} f(\beta)=\sum_{i = 1}^{n}\ln(1+\exp(-y_{i}x_{i}^{T}\beta))+\lambda\|\beta\|_{1}
\]
where \(x_{i}\) and \(y_{i}\) represent the \(i\)-th sample and its label respectively, and \(\lambda\) is the regularization parameter.

- Lipschitz continuity of the gradient:
\[
\|\nabla l(\beta+\Delta)-\nabla l(\beta)\|\leq\frac{1}{4}\lambda_{\max}(XX^{T})\|\Delta\|
\]

- Convergence rate theorem:
- For ISTA and the backtracking step - size search method:
\[
f(\beta_{k})-f(\beta^{*}) = O\left(\frac{1}{k}\right)
\]
- For FISTA:
\[
f(\beta_{k})-f(\beta^{*}) = O\left(\frac{1}{k^{2}\right)
\]

Through these improvements, this paper provides a more general and efficient solution for sparse logistic regression, especially when dealing with non - convex regularization terms.