Abstract:The least absolute shrinkage and selection operator (LASSO) of Tibshirani (1996) is a prominent estimator which selects significant (under some sense) features and kills insignificant ones. Indeed the LASSO shrinks features lager than a noise level to zero. In this paper, we force LASSO to be shrunken more by proposing a Stein-type shrinkage estimator emanating from the LASSO, namely the Stein-type LASSO. The newly proposed estimator proposes good performance in risk sense numerically. Variants of this estimator have smaller relative MSE and prediction error, compared to the LASSO, in the analysis of prostate cancer data set.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to improve the performance of the LASSO (Least Absolute Shrinkage and Selection Operator) estimator in order to obtain smaller risks and prediction errors. Specifically: 1. **Existing problems**: - LASSO is a method for feature selection and regression analysis. It achieves feature selection by introducing an L1 - regularization term to shrink coefficients and set them to zero. - Although LASSO performs well in many cases, in some cases, its risks (such as MSE) are still large, and the prediction error is high. 2. **Proposed new method**: - The author proposes a new estimator - Stein - type LASSO (SL), which further shrinks the LASSO estimates through double shrinking. - This method is based on Stein's shrinkage theory and aims to reduce the L2 risk of LASSO and improve the prediction performance. 3. **Specific objectives**: - Propose Stein - type LASSO and its variants (such as PRSL, that is, Positive Rule Stein - type LASSO), and verify their superiority through theoretical derivation and numerical experiments. - Conduct an empirical analysis on the prostate cancer data set to prove the effectiveness of the new method in practical applications. ### Main contributions - **Theoretical derivation**: By introducing the Stein - type shrinkage function, a new Stein - type LASSO estimator is constructed, and its superiority in L2 risk is proved. - **Numerical experiments**: Through Monte Carlo simulation studies, the performance of the new estimator and the traditional LASSO under different conditions is compared. The results show that the new method has smaller relative mean square error (RMSE) and prediction error in many cases. - **Empirical analysis**: An empirical analysis is carried out using the prostate cancer data set. The results show that Stein - type LASSO is superior to the traditional LASSO in prediction performance. ### Formula summary - L2 loss function: \[ L(\theta; \hat{\theta})=\|\hat{\theta}-\theta\|^{2} \] - L2 risk: \[ E[L(\theta; \hat{\theta})] \] - Stein - type LASSO estimator: \[ \hat{\beta}_{S}^{n}=\hat{\beta}_{L}^{n}+g(\hat{\beta}_{L}^{n}) \] - Positive Rule Stein - type LASSO (PRSL): \[ \hat{\beta}_{PRSL}^{n}=\left(\max \left(0,1-\frac{a}{W_{n}}\right) \hat{\beta}_{Lj}^{n} \mid j = 1,\ldots,p\right)^{\top} \] Through these improvements, the paper shows how to further improve the effects of feature selection and regression analysis on the basis of LASSO.

Double shrunken selection operator

Average case analysis of Lasso under ultra-sparse conditions

Regression Coefficient and Autoregressive Order Shrinkage and Selection Via the Lasso

Robust Regression Shrinkage and Consistent Variable Selection Through the LAD-Lasso

Improved LASSO

L1-Norm Least Squares Support Vector Regression Via the Alternating Direction Method of Multipliers

Post Selection Shrinkage Estimation for High Dimensional Data Analysis

Fused LASSO Penalized Least Absolute Deviation Estimator for High Dimensional Linear Regression

On an improvement of LASSO by scaling

Combinatorial Selection and Least Absolute Shrinkage via the CLASH Algorithm

Lasso Regression: Estimation and Shrinkage via Limit of Gibbs Sampling

Least squares after model selection in high-dimensional sparse models

Post‐Estimation Shrinkage in Full and Selected Linear Regression Models in Low‐Dimensional Data Revisited

Optimal Feature Selection for Sparse Linear Discriminant Analysis and Its Applications in Gene Expression Data

Safe Feature Elimination for the LASSO and Sparse Supervised Learning Problems

Robust Variable Selection Based on Relaxed Lad Lasso

Subset Selection with Shrinkage: Sparse Linear Modeling when the SNR is low

Shrinkage Tuning Parameter Selection with A Diverging Number of Parameters

Least sum of squares of trimmed residuals regression

On Mixture Regression Shrinkage and Selection via the MR-LASSO