Abstract:It is a common phenomenon that for high-dimensional and nonparametric statistical models, rate-optimal estimators balance squared bias and variance. Although this balancing is widely observed, little is known whether methods exist that could avoid the trade-off between bias and variance. We propose a general strategy to obtain lower bounds on the variance of any estimator with bias smaller than a prespecified bound. This shows to which extent the bias-variance trade-off is unavoidable and allows to quantify the loss of performance for methods that do not obey it. The approach is based on a number of abstract lower bounds for the variance involving the change of expectation with respect to different probability measures as well as information measures such as the Kullback-Leibler or $\chi^2$-divergence. In a second part of the article, the abstract lower bounds are applied to several statistical models including the Gaussian white noise model, a boundary estimation problem, the Gaussian sequence model and the high-dimensional linear regression model. For these specific statistical applications, different types of bias-variance trade-offs occur that vary considerably in their strength. For the trade-off between integrated squared bias and integrated variance in the Gaussian white noise model, we propose to combine the general strategy for lower bounds with a reduction technique. This allows us to reduce the original problem to a lower bound on the bias-variance trade-off for estimators with additional symmetry properties in a simpler statistical model. In the Gaussian sequence model, different phase transitions of the bias-variance trade-off occur. Although there is a non-trivial interplay between bias and variance, the rate of the squared bias and the variance do not have to be balanced in order to achieve the minimax estimation rate.

Towards A Mathematical Foundation Of Minimum-Variance Theory

The Minimum-Variance Theory Revisited

A Note on Minimum-Variance Theory and Beyond

Statistically Robust Estimation of Source Bearing Via Minimizing the Bhattacharyya Distance.

Study on Separation of Mechanical Vibration Based on the Principle of the Minimum Mutual Information

A NEW PERSPECTIVE ON ROBUST M-ESTIMATION: FINITE SAMPLE THEORY AND APPLICATIONS TO DEPENDENCE-ADJUSTED MULTIPLE TESTING

On a Theory of Precise Neural Control in a Noisy System

EIGENVALUE DISTRIBUTIONS OF VARIANCE COMPONENTS ESTIMATORS IN HIGH-DIMENSIONAL RANDOM EFFECTS MODELS

Three Laws of Multivariable Feedback Systems, Extended Spectral Flatness (Extended Wiener Entropy), 'Uncertainty Principles' in Variance Minimization, and Performance Limitations in Minimum Variance Estimation/Filtering

Statistical inference for rough volatility: Minimax Theory

Estimation and inference for minimizer and minimum of convex functions: optimality, adaptivity and uncertainty principles

The existence of the least favorable noise

Target motion analysis with unknown measurement noise variance

On Minimum Trace Factor Analysis -- An Old Song Sung to a New Tune

Maximum Principle for Mean-Field SDEs Under Model Uncertainty

Signal-to-noise ratio aware minimaxity and higher-order asymptotics

Minimum Variance Prediction of MIMO Linear Time-varying Systems

Linear Variance Bounds for Particle Approximations of Time-Homogeneous Feynman-Kac Formulae

On lower bounds for the bias-variance trade-off

Minimum Information Variability in Linear Langevin Systems via Model Predictive Control

Minimal Bounds on Nonlinearity in Auditory Processing