Abstract:A sparse modeling is a major topic in machine learning and statistics. LASSO (Least Absolute Shrinkage and Selection Operator) is a popular sparse modeling method while it has been known to yield unexpected large bias especially at a sparse representation. There have been several studies for improving this problem such as the introduction of non-convex regularization terms. The important point is that this bias problem directly affects model selection in applications since a sparse representation cannot be selected by a prediction error based model selection even if it is a good representation. In this article, we considered to improve this problem by introducing a scaling that expands LASSO estimator to compensate excessive shrinkage, thus a large bias in LASSO estimator. We here gave an empirical value for the amount of scaling. There are two advantages of this scaling method as follows. Since the proposed scaling value is calculated by using LASSO estimator, we only need LASSO estimator that is obtained by a fast and stable optimization procedure such as LARS (Least Angle Regression) under LASSO modification or coordinate descent. And, the simplicity of our scaling method enables us to derive SURE (Stein's Unbiased Risk Estimate) under the modified LASSO estimator with scaling. Our scaling method together with model selection based on SURE is fully empirical and do not need additional hyper-parameters. In a simple numerical example, we verified that our scaling method actually improves LASSO and the SURE based model selection criterion can stably choose an appropriate sparse model.

An Iterative Algorithm with Adaptive Weights and Sparse Laplacian Shrinkage for Regression Problems?

Adaptive Lasso in high-dimensional settings

A Constructive Approach to High-dimensional Regression

A constructive approach to L

A Variance Minimization Criterion to Feature Selection Using Laplacian Regularization

Penalized Sparse Covariance Regression with High Dimensional Covariates

Scaled sparse linear regression

An adaptive shortest-solution guided decimation approach to sparse high-dimensional linear regression

Regression Coefficient and Autoregressive Order Shrinkage and Selection Via the Lasso

Lasso Regression: Estimation and Shrinkage via Limit of Gibbs Sampling

Smooth Adjustment for Correlated Effects

Multivariate Regression Shrinkage and Selection by Canonical Correlation Analysis

Iterative Reweighted Framework Based Algorithms for Sparse Linear Regression with Generalized Elastic Net Penalty

Asymptotic Properties of Lasso+mLS and Lasso+Ridge in Sparse High-Dimensional Linear Regression

On an improvement of LASSO by scaling

Sparse Linear Regression With Missing Data

A Unified Dynamic Approach to Sparse Model Selection

High Dimensional Robust Sparse Regression

Weighted Lasso Estimates for Sparse Logistic Regression: Non-Asymptotic Properties with Measurement Errors

Enmsp: an elastic-net multi-step screening procedure for high-dimensional regression

Robust Regression Shrinkage and Consistent Variable Selection Through the LAD-Lasso