Abstract:In this article we study post-model selection estimators that apply ordinary least squares (OLS) to the model selected by first-step penalized estimators, typically Lasso. It is well known that Lasso can estimate the nonparametric regression function at nearly the oracle rate, and is thus hard to improve upon. We show that the OLS post-Lasso estimator performs at least as well as Lasso in terms of the rate of convergence, and has the advantage of a smaller bias. Remarkably, this performance occurs even if the Lasso-based model selection "fails" in the sense of missing some components of the "true" regression model. By the "true" model, we mean the best s-dimensional approximation to the nonparametric regression function chosen by the oracle. Furthermore, OLS post-Lasso estimator can perform strictly better than Lasso, in the sense of a strictly faster rate of convergence, if the Lasso-based model selection correctly includes all components of the "true" model as a subset and also achieves sufficient sparsity. In the extreme case, when Lasso perfectly selects the "true" model, the OLS post-Lasso estimator becomes the oracle estimator. An important ingredient in our analysis is a new sparsity bound on the dimension of the model selected by Lasso, which guarantees that this dimension is at most of the same order as the dimension of the "true" model. Our rate results are nonasymptotic and hold in both parametric and nonparametric models. Moreover, our analysis is not limited to the Lasso estimator acting as a selector in the first step, but also applies to any other estimator, for example, various forms of thresholded Lasso, with good rates and good sparsity properties. Our analysis covers both traditional thresholding and a new practical, data-driven thresholding scheme that induces additional sparsity subject to maintaining a certain goodness of fit. The latter scheme has theoretical guarantees similar to those of Lasso or OLS post-Lasso, but it dominates those procedures as well as traditional thresholding in a wide variety of experiments.

Evidence for Mate Choice by Male Guppies ( Poecilia reticulata )

Finite Sample Inference for Targeted Learning

Highly adaptive LASSO: Machine learning that provides valid nonparametric inference in realistic models

Wild bootstrap inference for penalized quantile regression for longitudinal data

Consistency of the bootstrap for asymptotically linear estimators based on machine learning

A Bootstrap Lasso + Partial Ridge Method to Construct Confidence Intervals for Parameters in High-dimensional Sparse Linear Models

The Lazy Bootstrap. A Fast Resampling Method for Evaluating Latent Class Model Fit

Bootstrap inference after using multiple queries for model selection

The nonparametric bootstrap for the current status model

Least squares after model selection in high-dimensional sparse models

An Easy Implementation of CV-TMLE

Collaborative-controlled LASSO for Constructing Propensity Score-based Estimators in High-Dimensional Data

Treatment bootstrapping: A new approach to quantify uncertainty of average treatment effect estimates

Bootstrap inference in functional linear regression models with scalar response under heteroscedasticity

Optimal Subsampling Bootstrap for Massive Data

Wild Residual Bootstrap Inference for Penalized Quantile Regression with Heteroscedastic Errors

Bootstrapping Inference of Average Treatment Effect in Completely Randomized Experiments with High-Dimensional Covariates

Statistical Inference with Local Optima

Estimating conditional hazard functions and densities with the highly-adaptive lasso

Collaborative targeted inference from continuously indexed nuisance parameter estimators

A simple bootstrap method for constructing nonparametric confidence bands for functions