Uniform asymptotic inference and the bootstrap after model selection

Ryan J Tibshirani, Alessandro Rinaldo, Rob Tibshirani, Larry Wasserman

2018-06-01

Abstract:Supplement to “Uniform asymptotic inference and the bootstrap after model selection”. This document gives additional figures, details, and proofs for the paper “Uniform asymptotic inference and the bootstrap after model selection.”.

What problem does this paper attempt to address?

Cluster-Robust Bootstrap Inference in Quantile Regression Models

Andreas Hagemann

DOI: https://doi.org/10.1080/01621459.2016.1148610

IF: 4.369

2017-01-02

Journal of the American Statistical Association

Abstract:In this article I develop a wild bootstrap procedure for cluster-robust inference in linear quantile regression models. I show that the bootstrap leads to asymptotically valid inference on the entire quantile regression process in a setting with a large number of small, heterogeneous clusters and provides consistent estimates of the asymptotic covariance function of that process. The proposed bootstrap procedure is easy to implement and performs well even when the number of clusters is much smaller than the sample size. An application to Project STAR data is provided. Supplementary materials for this article are available online.

statistics & probability
The Grid Bootstrap for Continuous Time Models

Yiu Lim Lui,Weilin Xiao,Jun Yu

DOI: https://doi.org/10.1080/07350015.2021.1930014

2021-01-01

Journal of Business and Economic Statistics

Abstract:This article proposes the new grid bootstrap to construct confidence intervals (CI) for the persistence parameter in a class of continuous-time models. It is different from the standard grid bootstrap of Hansen in dealing with the initial condition. The asymptotic validity of the CI is discussed under the in-fill scheme. The modified grid bootstrap leads to uniform inferences on the persistence parameter. Its improvement over in-fill asymptotics is achieved by expanding the coefficient-based statistic around its in-fill asymptotic distribution that is non-pivotal and depends on the initial condition. Monte Carlo studies show that the modified grid bootstrap performs better than Hansen's grid bootstrap. Empirical applications to the U.S. interest rates and volatilities suggest significant differences between the two bootstrap procedures when the initial condition is large.
An Introduction to the Bootstrap, Monogr. Stat. Appl. Probab., vol. 57

Bradley Efron, Robert J Tibshirani

1993-01-01

Abstract:
Bootstrap Inference for Panel Data Quantile Regression

Antonio F. Galvao,Thomas Parker,Zhijie Xiao

DOI: https://doi.org/10.1080/07350015.2023.2210189

2023-06-04

Journal of Business and Economic Statistics

Abstract:This article develops bootstrap methods for practical statistical inference in panel data quantile regression models with fixed effects. We consider random-weighted bootstrap resampling and formally establish its validity for asymptotic inference. The bootstrap algorithm is simple to implement in practice by using a weighted quantile regression estimation for fixed effects panel data. We provide results under conditions that allow for temporal dependence of observations within individuals, thus, encompassing a large class of possible empirical applications. Monte Carlo simulations provide numerical evidence the proposed bootstrap methods have correct finite sample properties. Finally, we provide an empirical illustration using the environmental Kuznets curve.

statistics & probability,social sciences, mathematical methods,economics
An introduction to the bootstrap

Bradley Efron, Robert J Tibshirani

1994-05-15

Abstract:An Introduction to the Bootstrap arms scientists and engineers as well as statisticians with the computational techniques they need to analyze and understand complicated data sets. The bootstrap is a computer-based method of statistical inference that answers statistical questions without formulas and gives a direct appreciation of variance, bias, coverage, and other probabilistic phenomena. This book presents an overview of the bootstrap and related methods for assessing statistical accuracy, concentrating on the ideas rather than their mathematical justification. Not just for beginners, the presentation starts off slowly, but builds in both scope and depth to ideas that are quite sophisticated.
Bootstrap inference after using multiple queries for model selection

Jelena Markovic,Jonathan Taylor

DOI: https://doi.org/10.48550/arXiv.1612.07811

2016-12-22

Methodology

Abstract:In this work, we provide a refinement of the selective CLT result of Tian and Taylor (2015), which allows for selective inference in non-parametric settings by adjusting for the asymptotic Gaussian limit for selection. Under some regularity assumptions on the density of the randomization, including heavier tails than Gaussian satisfied by e.g. logistic distribution, we prove the selective CLT holds without any assumptions on the underlying parameter, allowing for rare selection events. We also show a selective CLT result for Gaussian randomization, though the quantitative results are qualitatively different for the Gaussian randomization as compared to the heavier tailed results. Furthermore, we propose a bootstrap version of this test statistic, which is provably asymptotically pivotal uniformly across a family of non-parametric distributions. This result can be interpreted as resolving the impossibility results of Leeb and Potscher (2006). We describe several sampling methods involving the projected Langevin Monte Carlo to compute the bootstrapped test statistic and the corresponding confidence intervals valid after selection. The applications of our work include valid inferential and sampling tools after running various model selection algorithms including their combinations into multiple views/queries framework. We also present a way to do data carving, providing more powerful tests than classical data splitting by reusing the information in the data from the first stage.
Bootstrap Approximation of Model Selection Probabilities for Multimodel Inference Frameworks

Andres Dajles,Joseph Cavanaugh

DOI: https://doi.org/10.3390/e26070599

2024-07-15

Abstract:Most statistical modeling applications involve the consideration of a candidate collection of models based on various sets of explanatory variables. The candidate models may also differ in terms of the structural formulations for the systematic component and the posited probability distributions for the random component. A common practice is to use an information criterion to select a model from the collection that provides an optimal balance between fidelity to the data and parsimony. The analyst then typically proceeds as if the chosen model was the only model ever considered. However, such a practice fails to account for the variability inherent in the model selection process, which can lead to inappropriate inferential results and conclusions. In recent years, inferential methods have been proposed for multimodel frameworks that attempt to provide an appropriate accounting of modeling uncertainty. In the frequentist paradigm, such methods should ideally involve model selection probabilities, i.e., the relative frequencies of selection for each candidate model based on repeated sampling. Model selection probabilities can be conveniently approximated through bootstrapping. When the Akaike information criterion is employed, Akaike weights are also commonly used as a surrogate for selection probabilities. In this work, we show that the conventional bootstrap approach for approximating model selection probabilities is impacted by bias. We propose a simple correction to adjust for this bias. We also argue that Akaike weights do not provide adequate approximations for selection probabilities, although they do provide a crude gauge of model plausibility.
Model-free Bootstrap for a General Class of Stationary Time Series

Yiren Wang,Dimitris N. Politis

DOI: https://doi.org/10.48550/arXiv.1912.13185

2019-12-31

Abstract:A model-free bootstrap procedure for a general class of stationary time series is introduced. The theoretical framework is established, showing asymptotic validity of bootstrap confidence intervals for many statistics of interest. In addition, asymptotic validity of one-step ahead bootstrap prediction intervals is also demonstrated. Finite-sample experiments are conducted to empirically confirm the performance of the new method, and to compare with popular methods such as the block bootstrap and the autoregressive (AR)-sieve bootstrap.

Statistics Theory
An Introduction to Bootstrap Theory in Time Series Econometrics

Giuseppe Cavaliere,Heino Bohn Nielsen,Anders Rahbek

DOI: https://doi.org/10.2139/ssrn.3589144

2020-01-01

SSRN Electronic Journal

Abstract:This article provides an introduction to methods and challenges underlying application of the bootstrap in econometric modelling of economic and financial time series. Validity, or asymptotic validity, of the bootstrap is discussed as this is a key element in deciding whether the bootstrap is applicable in empirical contexts. That is, as detailed here, bootstrap validity relies on regularity conditions, which need to be verified on a case-by-case basis. To fix ideas, asymptotic validity is discussed in terms of the leading example of bootstrap-based hypothesis testing in the well-known first order auto-regressive model. In particular, bootstrap versions of classic convergence in probability and distribution, and hence of laws of large numbers and central limit theorems, are discussed as crucial ingredients to establish bootstrap validity. Regularity conditions and their implications for possible improvements in terms of (empirical) size and power for bootstrap-based testing, when compared to asymptotic testing, are illustrated by simulations. Following this, an overview of selected recent advances in the application of bootstrap methods in econometrics is also given.
The Limit Distribution of the Bootstrap for the Unit Root Test Statistic when the Residuals Are Dependent

Li-Xin Zhang,Xiao-Rong Yang

DOI: https://doi.org/10.1007/s00184-006-0070-y

IF: 0.96

2006-01-01

Metrika

Abstract:In this article, the unit root test for the AR(1) model with dependent residuals is considered. We adopt a bootstrap procedure to bootstrap the residuals with bootstrap sample size m less than the size n of the original sample. Under the assumptions that m → ∞ and m/n → 0, the convergence in probability of the bootstrap distribution function is established.
Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy

Bradley Efron, Robert Tibshirani

1986-02-01

Abstract:This is a review of bootstrap methods, concentrating on basic ideas and applications rather than theoretical considerations. It begins with an exposition of the bootstrap estimate of standard error for one-sample situations. Several examples, some involving quite complicated statistical procedures, are given. The bootstrap is then extended to other measures of statistical accuracy such as bias and prediction error, and to complicated data structures such as time series, censored data, and regression models. Several more examples are presented illustrating these ideas. The last third of the paper deals mainly with bootstrap confidence intervals.
Hybrid Confidence Intervals for Informative Uniform Asymptotic Inference After Model Selection

Adam McCloskey

DOI: https://doi.org/10.48550/arXiv.2011.12873

2021-11-24

Abstract:I propose a new type of confidence interval for correct asymptotic inference after using data to select a model of interest without assuming any model is correctly specified. This hybrid confidence interval is constructed by combining techniques from the selective inference and post-selection inference literatures to yield a short confidence interval across a wide range of data realizations. I show that hybrid confidence intervals have correct asymptotic coverage, uniformly over a large class of probability distributions that do not bound scaled model parameters. I illustrate the use of these confidence intervals in the problem of inference after using the LASSO objective function to select a regression model of interest and provide evidence of their desirable length and coverage properties in small samples via a set of Monte Carlo experiments that entail a variety of different data distributions as well as an empirical application to the predictors of diabetes disease progression.

Methodology,Statistics Theory
Selective inference after feature selection via multiscale bootstrap

Yoshikazu Terada,Hidetoshi Shimodaira

DOI: https://doi.org/10.48550/arXiv.1905.10573

2022-06-01

Abstract:It is common to show the confidence intervals or $p$-values of selected features, or predictor variables in regression, but they often involve selection bias. The selective inference approach solves this bias by conditioning on the selection event. Most existing studies of selective inference consider a specific algorithm, such as Lasso, for feature selection, and thus they have difficulties in handling more complicated algorithms. Moreover, existing studies often consider unnecessarily restrictive events, leading to over-conditioning and lower statistical power. Our novel and widely-applicable resampling method via multiscale bootstrap addresses these issues to compute an approximately unbiased selective $p$-value for the selected features. As a simplification of the proposed method, we also develop a simpler method via the classical bootstrap. We prove that the $p$-value computed by our multiscale bootstrap method is more accurate than the classical bootstrap method. Furthermore, numerical experiments demonstrate that our algorithm works well even for more complicated feature selection methods such as non-convex regularization.

Methodology
The Adequate Bootstrap

Toby Kenney,Hong Gu

DOI: https://doi.org/10.48550/arXiv.1608.05913

2016-08-21

Abstract:There is a fundamental disconnect between what is tested in a model adequacy test, and what we would like to test. The usual approach is to test the null hypothesis "Model M is the true model." However, Model M is never the true model. A model might still be useful even if we have enough data to reject it. In this paper, we present a technique to assess the adequacy of a model from the philosophical standpoint that we know the model is not true, but we want to know if it is useful. Our solution to this problem is to measure the parameter uncertainty in our estimates caused by the model uncertainty. We use bootstrap inference on samples of a smaller size, for which the model cannot be rejected. We use a model adequacy test to choose a bootstrap size with limited probability of rejecting the model and perform inference for samples of this size based on a nonparametric bootstrap. Our idea is that if we base our inference on a sample size at which we do not reject the model, then we should be happy with this inference, because we would have been confident in it if our original dataset had been this size.

Methodology
Bootstrap for inference after model selection and model averaging for likelihood models

Andrea C. Garcia-Angulo,Gerda Claeskens

DOI: https://doi.org/10.1007/s00184-024-00956-2

IF: 0.96

2024-03-06

Metrika

Abstract:A one-step semiparametric bootstrap procedure is constructed to estimate the distribution of estimators after model selection and of model averaging estimators with data-dependent weights. The method is generally applicable to non-normal models. Misspecification is allowed for all candidate parametric models. The semiparametric bootstrap estimator is shown to be consistent within specific regions such that the good and the bad candidate models are separated. Simulation studies exemplify that the bootstrap procedure leads to short confidence intervals with a good coverage.

statistics & probability
Bootstrap confidence intervals and bootstrap approximations

Thomas DiCiccio, Robert Tibshirani

1987-03-01

Abstract:The BCa bootstrap procedure (Efron 1987) for constructing parametric and nonparametric confidence intervals is considered. Like the bootstrap, this procedure can be applied to complicated problems in a wide range of situations. For models indexed by a scalar parameter θ with efficient estimator , the BCa procedure relies on the existence of a transformation g(·) such that () is approximately normally distributed with standard deviation 1 + ag(θ), although explicit knowledge of g(·) is not required. In this article, we show how to construct this transformation by generalizing the one given by Efron (1987, sec. 10) for translation families. This construction consists of the composition of a variance-stabilizing transformation and a skewness-reducing transformation. It produces a new interval, the BCa interval, that is asymptotically equivalent to the BCa interval and can be computed without bootstrap sampling. We also …
The Bootstrap Method for Assessing Statistical Accuracy

Efron and Tibshirani

1985-01-01

Abstract:
An Introduction to the Bootstrap, volume 57 of

Bradley Efron, Robert J Tibshirani

1993-01-01

Abstract:
A novel bootstrap goodness-of-fit test for normal linear regression models

Scott H. Koeneman,Joseph E. Cavanaugh

DOI: https://doi.org/10.1007/s10182-024-00517-y

2024-11-20

AStA Advances in Statistical Analysis

Abstract:In this work, the distributional properties of the goodness-of-fit term in likelihood-based information criteria are explored. These properties are then leveraged to construct a novel goodness-of-fit test for normal linear regression models that relies on a nonparametric bootstrap. Several simulation studies are performed to investigate the properties and efficacy of the developed procedure, with these studies demonstrating that the bootstrap test offers distinct advantages as compared to other methods of assessing the goodness-of-fit of a normal linear regression model. Our inferential technique can be employed using the DBModelSelect R package, available freely via the Comprehensive R Archive Network.

statistics & probability
Asymptotic Properties of the Bootstrap Unit Root Test Statistic under Possibly Infinite Variance

Ke-Ang Fu,Jie Li,Xiao-Rong Yang

DOI: https://doi.org/10.1080/03610926.2014.901362

2016-01-01

Abstract:In this article, the unit root test for the AR(1) model is discussed, under the condition that the innovations of the model are in the domain of attraction of the normal law with possibly infinite variances. By using residual bootstrap with sample size m < n (n being the size of the original sample), we bootstrap the least-squares estimator of the autoregressive parameter. Under some mild assumptions, we prove that the null distribution of the unit root test statistic based on the least-square estimator of the autoregressive parameter can be approximated by using residual bootstrap.

Uniform asymptotic inference and the bootstrap after model selection

Cluster-Robust Bootstrap Inference in Quantile Regression Models

The Grid Bootstrap for Continuous Time Models

An Introduction to the Bootstrap, Monogr. Stat. Appl. Probab., vol. 57

Bootstrap Inference for Panel Data Quantile Regression

An introduction to the bootstrap

Bootstrap inference after using multiple queries for model selection

Bootstrap Approximation of Model Selection Probabilities for Multimodel Inference Frameworks

Model-free Bootstrap for a General Class of Stationary Time Series

An Introduction to Bootstrap Theory in Time Series Econometrics

The Limit Distribution of the Bootstrap for the Unit Root Test Statistic when the Residuals Are Dependent

Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy

Hybrid Confidence Intervals for Informative Uniform Asymptotic Inference After Model Selection

Selective inference after feature selection via multiscale bootstrap

The Adequate Bootstrap

Bootstrap for inference after model selection and model averaging for likelihood models

Bootstrap confidence intervals and bootstrap approximations

The Bootstrap Method for Assessing Statistical Accuracy

An Introduction to the Bootstrap, volume 57 of

A novel bootstrap goodness-of-fit test for normal linear regression models

Asymptotic Properties of the Bootstrap Unit Root Test Statistic under Possibly Infinite Variance