Abstract:There is growing interest in improving our algorithmic understanding of fundamental statistical problems such as mean estimation, driven by the goal of understanding the limits of what we can extract from valuable data. The state of the art results for mean estimation in $\mathbb{R}$ are 1) the optimal sub-Gaussian mean estimator by [LV22], with the tight sub-Gaussian constant for all distributions with finite but unknown variance, and 2) the analysis of the median-of-means algorithm by [BCL13] and a lower bound by [DLLO16], characterizing the big-O optimal errors for distributions for which only a $1+\alpha$ moment exists for $\alpha \in (0,1)$. Both results, however, are optimal only in the worst case. We initiate the fine-grained study of the mean estimation problem: Can algorithms leverage useful features of the input distribution to beat the sub-Gaussian rate, without explicit knowledge of such features? We resolve this question with an unexpectedly nuanced answer: "Yes in limited regimes, but in general no". For any distribution $p$ with a finite mean, we construct a distribution $q$ whose mean is well-separated from $p$'s, yet $p$ and $q$ are not distinguishable with high probability, and $q$ further preserves $p$'s moments up to constants. The main consequence is that no reasonable estimator can asymptotically achieve better than the sub-Gaussian error rate for any distribution, matching the worst-case result of [LV22]. More generally, we introduce a new definitional framework to analyze the fine-grained optimality of algorithms, which we call "neighborhood optimality", interpolating between the unattainably strong "instance optimality" and the trivially weak "admissibility" definitions. Applying the new framework, we show that median-of-means is neighborhood optimal, up to constant factors. It is open to find a neighborhood-optimal estimator without constant factor slackness.

An Axiomatisation of Error Intolerant Estimation

Risk-averse estimation, an axiomatic approach to inference, and Wallace-Freeman without MML

Axioms of adaptivity

Data-driven Error Estimation: Upper Bounding Multiple Errors with No Technical Debt

A New Reliability Method Combining Adaptive Kriging and Active Variance Reduction Using Multiple Importance Sampling

Robust Bayesian Inference for Berkson and Classical Measurement Error Models

On the optimality of DIA-estimators: theory and applications

Optimal estimator for uncertainty-based measurement quality control

Minimum Total Error Entropy Method for Parameter Estimation

A NEW PERSPECTIVE ON ROBUST M-ESTIMATION: FINITE SAMPLE THEORY AND APPLICATIONS TO DEPENDENCE-ADJUSTED MULTIPLE TESTING

A general algorithm for error-in-variables regression modelling using Monte Carlo expectation maximization

A taxonomy of estimator consistency on discrete estimation problems

Optimal Error Regions For Quantum State Estimation

Optimality in Mean Estimation: Beyond Worst-Case, Beyond Sub-Gaussian, and Beyond $1+α$ Moments

Asymptotic efficiency of inferential models and a possibilistic Bernstein--von Mises theorem

Towards a Law of Iterated Expectations for Heuristic Estimators

A Heisenberg-esque Uncertainty Principle for Simultaneous (Machine) Learning and Error Assessment?

The estimation error of general first order methods

On Optimal Estimations with Minimum Error Entropy Criterion

Precise Error Analysis of Regularized M-estimators in High-dimensions

Adaptive importance sampling via minimization of estimators of cross-entropy, mean square, and inefficiency constant