Abstract:Cross validation (CV) and the bootstrap are ubiquitous model-agnostic tools for assessing the error or variability of machine learning and statistical estimators. However, these methods require repeatedly re-fitting the model with different weighted versions of the original dataset, which can be prohibitively time-consuming. For sufficiently regular optimization problems the optimum depends smoothly on the data weights, and so the process of repeatedly re-fitting can be approximated with a Taylor series that can be often evaluated relatively quickly. The first-order approximation is known as the "infinitesimal jackknife" in the statistics literature and has been the subject of recent interest in machine learning for approximate CV. In this work, we consider high-order approximations, which we call the "higher-order infinitesimal jackknife" (HOIJ). Under mild regularity conditions, we provide a simple recursive procedure to compute approximations of all orders with finite-sample accuracy bounds. Additionally, we show that the HOIJ can be efficiently computed even in high dimensions using forward-mode automatic differentiation. We show that a linear approximation with bootstrap weights approximation is equivalent to those provided by asymptotic normal approximations. Consequently, the HOIJ opens up the possibility of enjoying higher-order accuracy properties of the bootstrap using local approximations. Consistency of the HOIJ for leave-one-out CV under different asymptotic regimes follows as corollaries from our finite-sample bounds under additional regularity assumptions. The generality of the computation and bounds motivate the name "higher-order Swiss Army infinitesimal jackknife."

Acceleration for the mu+ mu- Collider

Asymptotic equivalence of the jackknife and infinitesimal jackknife variance estimators for some smooth statistics

The Bayesian Infinitesimal Jackknife for Variance

A Higher-Order Swiss Army Infinitesimal Jackknife

Precise Asymptotics in the Law of Logarithm for U-statistics

When is it worthwhile to jackknife? Breaking the quadratic barrier for Z-estimators

Fast and Reliable Jackknife and Bootstrap Methods for Cluster‐Robust Inference

Predictive inference with the jackknife+

Jackknife Method for Intermediate Quantiles

Consistency of the bootstrap for asymptotically linear estimators based on machine learning

A Comparison of Resampling and Recursive Partitioning Methods in Random Forest for Estimating the Asymptotic Variance Using the Infinitesimal Jackknife

Discriminative Jackknife: Quantifying Uncertainty in Deep Learning via Higher-Order Influence Functions

Asymptotic analysis of locally weighted jackknife prediction

Asymptotic Distribution Theory for Nonparametric Entropy Measures of Serial Dependence

Precise Asymptotics for the First Moment of the Error Variance Estimator in Linear Models

Asymptotics for regression models under loss of identifiability

Jackknife Model Averaging for Mixed-Data Kernel-Weighted Spline Quantile Regressions

Deterministic Inequalities for Smooth M-estimators

A review of asymptotic theory of estimating functions

Analysis of Bootstrap and Subsampling in High-dimensional Regularized Regression

On the Variance of Single-Run Unbiased Stochastic Derivative Estimators.