Abstract:The elastic net penalty is frequently employed in high-dimensional statistics for parameter regression and variable selection. It is particularly beneficial compared to lasso when the number of predictors greatly surpasses the number of observations. However, empirical evidence has shown that the $\ell_q$-norm penalty (where $0 < q < 1$) often provides better regression compared to the $\ell_1$-norm penalty, demonstrating enhanced robustness in various scenarios. In this paper, we explore a generalized elastic net model that employs a $\ell_r$-norm (where $r \geq 1$) in loss function to accommodate various types of noise, and employs a $\ell_q$-norm (where $0 < q < 1$) to replace the $\ell_1$-norm in elastic net penalty. Theoretically, we establish the computable lower bounds for the nonzero entries of the generalized first-order stationary points of the proposed generalized elastic net model. For implementation, we develop two efficient algorithms based on the locally Lipschitz continuous $\epsilon$-approximation to $\ell_q$-norm. The first algorithm employs an alternating direction method of multipliers (ADMM), while the second utilizes a proximal majorization-minimization method (PMM), where the subproblems are addressed using the semismooth Newton method (SNN). We also perform extensive numerical experiments with both simulated and real data, showing that both algorithms demonstrate superior performance. Notably, the PMM-SSN is efficient than ADMM, even though the latter provides a simpler implementation.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in high - dimensional sparse linear regression, how to improve the existing Elastic Net model to enhance regression performance and robustness. Specifically, the authors propose a generalized Elastic Net model, which uses the $\ell_r$-norm ($r\geq1$) as a loss function to adapt to different types of noise, and replaces the traditional $\ell_1$-norm with the $\ell_q$-norm ($0 < q < 1$) as a penalty term. Through this method, they hope to obtain better regression results when dealing with non - normal distribution noise and other complex situations. ### Specific Problem Description 1. **Limitations of Existing Methods**: - Lasso and the traditional Elastic Net model have limitations when dealing with high - dimensional data, especially when the number of predictor variables is much larger than the number of observed samples. - Although the $\ell_1$-norm can achieve continuous shrinkage and automatic variable selection, it can lead to over - regularization in some cases and has limited ability to select grouped variables. - When the noise does not follow a normal distribution, the least - squares estimate may no longer be applicable, and a more robust loss function is required. 2. **Objectives of Improvement**: - Replace the $\ell_1$-norm with the $\ell_q$-norm ($0 < q < 1$) to improve sparsity and robustness. - Introduce the $\ell_r$-norm ($r\geq1$) as a loss function to adapt to different types of noise (such as heavy - tailed noise, uniformly distributed noise, etc.). - Develop effective algorithms to solve this new optimization problem. ### Main Contributions of the Paper - **Theoretical Analysis**: Establish the lower bound of the generalized first - order stationary point and prove that the local minimum points satisfy these conditions. - **Algorithm Design**: Propose two efficient algorithms - one based on the Alternating Direction Method of Multipliers (ADMM) and the other based on the Proximal Max - Min Method (PMM) combined with the Semi - smooth Newton Method (SNN), for solving the generalized Elastic Net model. - **Numerical Experiments**: Verify the effectiveness of the new model and its algorithms through simulation and real - data, especially showing better performance than existing methods when dealing with different types of noise. ### Summary This paper aims to solve the over - regularization and noise - adaptability problems in high - dimensional sparse linear regression by introducing the generalized Elastic Net model and developing corresponding efficient algorithms, thereby improving regression performance and robustness.

Iterative Reweighted Framework Based Algorithms for Sparse Linear Regression with Generalized Elastic Net Penalty

Elastic Net with Adaptive Weight for Image Denoising

A new penalized least absolute deviation model for high dimensional sparse linear regression and an efficient sequential linear programming algorithm

Sparse Estimation Via Lower-Order Penalty Optimization Methods in High-Dimensional Linear Regression.

Stability of the Elastic Net Estimator

Robust Elastic Net Regression

Enmsp: an elastic-net multi-step screening procedure for high-dimensional regression

A New Penalized Recursive Least Squares Method With a Variable Regularization Factor for Adaptive Sparse Filtering.

A Generalized Linear Joint Trained Framework for Semi-Supervised Learning of Sparse Features

Robust and sparse estimation methods for high dimensional linear and logistic regression

An Extended Newton-type Algorithm for $\ell_2$-Regularized Sparse Logistic Regression and Its Efficiency for Classifying Large-scale Datasets

Distributed Sparse Recursive Least-Squares over Networks

Scaled sparse linear regression

Generalized Low-rank plus Sparse Tensor Estimation by Fast Riemannian Optimization

The Trimmed Lasso: Sparsity and Robustness

A Semismooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning

Discriminative Elastic-Net Regularized Linear Regression.

Elastic Gradient Descent, an Iterative Optimization Method Approximating the Solution Paths of the Elastic Net

Proximal Iteration for Nonlinear Adaptive Lasso

Penalized robust estimators in logistic regression with applications to sparse models

Reduced-Space Iteratively Reweighted Second-Order Methods for Nonconvex Sparse Regularization