Abstract:In this article we study post-model selection estimators that apply ordinary least squares (OLS) to the model selected by first-step penalized estimators, typically Lasso. It is well known that Lasso can estimate the nonparametric regression function at nearly the oracle rate, and is thus hard to improve upon. We show that the OLS post-Lasso estimator performs at least as well as Lasso in terms of the rate of convergence, and has the advantage of a smaller bias. Remarkably, this performance occurs even if the Lasso-based model selection "fails" in the sense of missing some components of the "true" regression model. By the "true" model, we mean the best s-dimensional approximation to the nonparametric regression function chosen by the oracle. Furthermore, OLS post-Lasso estimator can perform strictly better than Lasso, in the sense of a strictly faster rate of convergence, if the Lasso-based model selection correctly includes all components of the "true" model as a subset and also achieves sufficient sparsity. In the extreme case, when Lasso perfectly selects the "true" model, the OLS post-Lasso estimator becomes the oracle estimator. An important ingredient in our analysis is a new sparsity bound on the dimension of the model selected by Lasso, which guarantees that this dimension is at most of the same order as the dimension of the "true" model. Our rate results are nonasymptotic and hold in both parametric and nonparametric models. Moreover, our analysis is not limited to the Lasso estimator acting as a selector in the first step, but also applies to any other estimator, for example, various forms of thresholded Lasso, with good rates and good sparsity properties. Our analysis covers both traditional thresholding and a new practical, data-driven thresholding scheme that induces additional sparsity subject to maintaining a certain goodness of fit. The latter scheme has theoretical guarantees similar to those of Lasso or OLS post-Lasso, but it dominates those procedures as well as traditional thresholding in a wide variety of experiments.

Non-asymptotic model selection for linear non least-squares estimation in regression models and inverse problems

Regularization and variable selection for infinite variance autoregressive models

Adaptive estimation of a quadratic functional by model selection

Two Stage Non-penalized Corrected Least Squares for High Dimensional Linear Models with Measurement error or Missing Covariates

Bayesian variable selection in linear regression models with instrumental variables

Model selection in logistic regression

Least squares after model selection in high-dimensional sparse models

Nonlinear Recursive Minimum Model Error Estimation

Adaptive Covariance Estimation with model selection

A Nonprobabilistic Approach for Linear Model Selection

An accelerated inexact Newton regularization scheme with a learned feature-selection rule for non-linear inverse problems

An efficient optimization approach for best subset selection in linear regression, with application to model selection and fitting in autoregressive time-series

Non-Concave Penalization in Linear Mixed-Effects Models and Regularized Selection of Fixed Effects

Non-intrusive Least-Squares Functional A Posteriori Error Estimator: Linear and Nonlinear Problems with Plain Convergence

Model selection for density estimation with L2-loss

Generalized Bayes Approach to Inverse Problems with Model Misspecification

Data-driven calibration of linear estimators with minimal penalties

Least Squares Approximations in Linear Statistical Inverse Learning Problems

Consistent Variable Selection for a Nonparametric Nonlinear System by Inverse and Contour Regressions.

Non-convex Penalized Estimation in High-Dimensional Models with Single-Index Structure

Non-Bayesian Post-Model-Selection Estimation as Estimation Under Model Misspecification