Abstract:It is a common phenomenon that for high-dimensional and nonparametric statistical models, rate-optimal estimators balance squared bias and variance. Although this balancing is widely observed, little is known whether methods exist that could avoid the trade-off between bias and variance. We propose a general strategy to obtain lower bounds on the variance of any estimator with bias smaller than a prespecified bound. This shows to which extent the bias-variance trade-off is unavoidable and allows to quantify the loss of performance for methods that do not obey it. The approach is based on a number of abstract lower bounds for the variance involving the change of expectation with respect to different probability measures as well as information measures such as the Kullback-Leibler or $\chi^2$-divergence. In a second part of the article, the abstract lower bounds are applied to several statistical models including the Gaussian white noise model, a boundary estimation problem, the Gaussian sequence model and the high-dimensional linear regression model. For these specific statistical applications, different types of bias-variance trade-offs occur that vary considerably in their strength. For the trade-off between integrated squared bias and integrated variance in the Gaussian white noise model, we propose to combine the general strategy for lower bounds with a reduction technique. This allows us to reduce the original problem to a lower bound on the bias-variance trade-off for estimators with additional symmetry properties in a simpler statistical model. In the Gaussian sequence model, different phase transitions of the bias-variance trade-off occur. Although there is a non-trivial interplay between bias and variance, the rate of the squared bias and the variance do not have to be balanced in order to achieve the minimax estimation rate.

Variance Bounds: Some Old and Some New

The Variance of Partial Sums of Strong Near-Epoch Dependent Variables

Remark On Variance Bounds

A variance bound for a general function of independent noncommutative random variables

LIL and the Approximation of Rectangular Sums of B-valued Random Variables when Extreme Terms Are Excluded

On the Variance of Average Distance of Subsets in the Hamming Space

Limit Theorems for Dependent Bernoulli Variables with Statistical Inference

Variance bound of function of order statistics

Lower bounds for variances of Poisson functionals

The Variance Conjecture on Some Polytopes

A variance-sensitive Gaussian concentration inequality

Finite de Finetti bounds in relative entropy

Covariance Identities and Variance Bounds for Infinitely Divisible Random Variables and Their Applications

Variance Inequalities for Transformed Fréchet Means in Hadamard Spaces

On lower bounds for the bias-variance trade-off

Asymptotic Expansions and two-sided Bounds in Randomized Central Limit Theorems

Bernoulli sums and Rényi entropy inequalities

On Some Extensions of the Boué-Dupuis Variational Formula

The variance of integers without small prime factors in short intervals

Deviation bounds for the norm of a random vector under exponential moment conditions with applications