Abstract:Bagging an estimator approximately doubles its bias through the impact of bagging on quadratic terms in expansions of the estimator. This dicult y can be alleviated by bagging a suitably bias-corrected estimator, however. In these and other circumstances, what is the overall impact of bagging and/or bias cor- rection, and how can it be characterised? We answer these questions in the case of general estimators dened by estimating equations, including for example max- imum likelihood and method of moments estimators. It is shown that, despite the considerable variety of estimators that can be constructed by bagging and bias cor- rection, the number of modes of behaviour is very small. In particular, bagging a bias-corrected estimator produces a new estimator that is second-order equivalent to the original, unadjusted estimator. Furthermore, the conventional bagged es- timator, and the standard bias-corrected estimator, represent virtually equal but opposite adjustments of the conventional estimator. In particular, bagging adds back the adjustment provided by bias correction. If we bag a doubly bias corrected estimator, constructed so as to counteract the tendency of bagging to exacerbate bias, then the result is an estimator that is second-order equivalent to the standard bias-corrected estimator. These results do not depend on the manner of bias cor- rection; that procedure may be implemented using the jackknife, the parametric bootstrap or the nonparametric bootstrap. They show that, when bagging is ap- plied to relatively conventional statistical problems, it cannot reliably be expected to improve performance. Its domain is, in eect, restricted to problems such as regression trees, where variability is so high that it cannot be plausibly modelled using the approach taken here.

Correction of overfitting bias in regression models

Bias Reduction in Instrumental Variable Estimation through First-Stage Shrinkage

Bias correction of posterior means using MCMC outputs

Bias Correction and Robust Inference in Semiparametric Models

Inference for High-Dimensional Linear Expectile Regression with De-Biasing Method

Empirical bias-reducing adjustments to estimating functions

When is it worthwhile to jackknife? Breaking the quadratic barrier for Z-estimators

Bias correction for multiple covariate analysis using empirical bayesian estimation in mixed-effects models for longitudinal data

Effects of Bagging and Bias Correction on Estimators Defined by Estimating Equations

Efficient estimation and correction of selection-induced bias with order statistics

Combining matching and linear regression: Introducing a mathematical framework and software for simulations, diagnostics and calibration

Root-n consistent semiparametric learning with high-dimensional nuisance functions under minimal sparsity

Estimation for Biased Partial Linear Single Index Models

Jeffreys-prior penalty for high-dimensional logistic regression: A conjecture about aggregate bias

Double Descent: Understanding Linear Model Estimation of Nonidentifiable Parameters and a Model for Overfitting

Covariate-adjusted nonlinear regression

Eliminating Ratio Bias for Gradient-based Simulated Parameter Estimation

A simulation-based estimation method for bias reduction

A constrained maximum likelihood approach to developing well-calibrated models for predicting binary outcomes

Unraveling overoptimism and publication bias in ML-driven science

The Bayesian Infinitesimal Jackknife for Variance