Abstract:In this article we study post-model selection estimators that apply ordinary least squares (OLS) to the model selected by first-step penalized estimators, typically Lasso. It is well known that Lasso can estimate the nonparametric regression function at nearly the oracle rate, and is thus hard to improve upon. We show that the OLS post-Lasso estimator performs at least as well as Lasso in terms of the rate of convergence, and has the advantage of a smaller bias. Remarkably, this performance occurs even if the Lasso-based model selection "fails" in the sense of missing some components of the "true" regression model. By the "true" model, we mean the best s-dimensional approximation to the nonparametric regression function chosen by the oracle. Furthermore, OLS post-Lasso estimator can perform strictly better than Lasso, in the sense of a strictly faster rate of convergence, if the Lasso-based model selection correctly includes all components of the "true" model as a subset and also achieves sufficient sparsity. In the extreme case, when Lasso perfectly selects the "true" model, the OLS post-Lasso estimator becomes the oracle estimator. An important ingredient in our analysis is a new sparsity bound on the dimension of the model selected by Lasso, which guarantees that this dimension is at most of the same order as the dimension of the "true" model. Our rate results are nonasymptotic and hold in both parametric and nonparametric models. Moreover, our analysis is not limited to the Lasso estimator acting as a selector in the first step, but also applies to any other estimator, for example, various forms of thresholded Lasso, with good rates and good sparsity properties. Our analysis covers both traditional thresholding and a new practical, data-driven thresholding scheme that induces additional sparsity subject to maintaining a certain goodness of fit. The latter scheme has theoretical guarantees similar to those of Lasso or OLS post-Lasso, but it dominates those procedures as well as traditional thresholding in a wide variety of experiments.

Rates of Convergence of the Adaptive Elastic Net and the Post-Selection Procedure in Ultra-High Dimensional Sparse Models

Adaptive Lasso in high-dimensional settings

Nonnegative Adaptive Lasso for Ultra-High Dimensional Regression Models and A Two-Stage Method Applied in Financial Modeling

Consistent Tuning Parameter Selection in High Dimensional Sparse Linear Regression

Asymptotic Properties of Lasso+mLS and Lasso+Ridge in Sparse High-Dimensional Linear Regression

Post Selection Shrinkage Estimation for High Dimensional Data Analysis

High-dimensional variable selection with heterogeneous signals: A precise asymptotic perspective

Least squares after model selection in high-dimensional sparse models

Elastic Net Procedure for Partially Linear Models

Stability of the Elastic Net Estimator

High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking

Enmsp: an elastic-net multi-step screening procedure for high-dimensional regression

Adaptive posterior concentration rates for sparse high-dimensional linear regression with random design and unknown error variance

Sequential Lasso Cum EBIC for Feature Selection with Ultra-High Dimensional Feature Space

Prediction Weighted Maximum Frequency Selection

Variable selection using axis-aligned random projections for partial least-squares regression

Model selection via standard error adjusted adaptive lasso

Variable Selection via Adaptive False Negative Control in Linear Regression

A Semismooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning

The EAS approach to variable selection for multivariate response data in high-dimensional settings

Consistent group selection in high-dimensional linear regression