Abstract:There is growing interest in improving our algorithmic understanding of fundamental statistical problems such as mean estimation, driven by the goal of understanding the limits of what we can extract from valuable data. The state of the art results for mean estimation in $\mathbb{R}$ are 1) the optimal sub-Gaussian mean estimator by [LV22], with the tight sub-Gaussian constant for all distributions with finite but unknown variance, and 2) the analysis of the median-of-means algorithm by [BCL13] and a lower bound by [DLLO16], characterizing the big-O optimal errors for distributions for which only a $1+\alpha$ moment exists for $\alpha \in (0,1)$. Both results, however, are optimal only in the worst case. We initiate the fine-grained study of the mean estimation problem: Can algorithms leverage useful features of the input distribution to beat the sub-Gaussian rate, without explicit knowledge of such features? We resolve this question with an unexpectedly nuanced answer: "Yes in limited regimes, but in general no". For any distribution $p$ with a finite mean, we construct a distribution $q$ whose mean is well-separated from $p$'s, yet $p$ and $q$ are not distinguishable with high probability, and $q$ further preserves $p$'s moments up to constants. The main consequence is that no reasonable estimator can asymptotically achieve better than the sub-Gaussian error rate for any distribution, matching the worst-case result of [LV22]. More generally, we introduce a new definitional framework to analyze the fine-grained optimality of algorithms, which we call "neighborhood optimality", interpolating between the unattainably strong "instance optimality" and the trivially weak "admissibility" definitions. Applying the new framework, we show that median-of-means is neighborhood optimal, up to constant factors. It is open to find a neighborhood-optimal estimator without constant factor slackness.

On Optimal Estimations with Minimum Error Entropy Criterion

A New Interpretation on the MMSE As a Robust MEE Criterion

Minimum Total Error Entropy Method for Parameter Estimation

Restricted Minimum Error Entropy Criterion for Robust Classification

Robust Adaptive Filtering based on M-estimation-based Minimum Error Entropy Criterion

Consistency analysis of an empirical minimum error entropy algorithm

Generalized Minimum Error with Fiducial Points Criterion for Robust Learning

Fixed-Point Minimum Error Entropy With Fiducial Points

Optimum estimation via gradients of partition functions and information measures: a statistical-mechanical perspective

A NEW PERSPECTIVE ON ROBUST M-ESTIMATION: FINITE SAMPLE THEORY AND APPLICATIONS TO DEPENDENCE-ADJUSTED MULTIPLE TESTING

A Bilateral Bound on the Mean-Square Error for Estimation in Model Mismatch

Precise Error Analysis of Regularized M-estimators in High-dimensions

On the Equivalence of Maximum SNR and MMSE Estimation: Applications to Additive Non-Gaussian Channels and Quantized Observations

Δ-Entropy: Definition, Properties and Applications in System Identification with Quantized Data

Mixtures of Gaussians and Minimum Relative Entropy Techniques for Modeling Continuous Uncertainties

Optimal convex $M$-estimation via score matching

Existence of solutions to the nonlinear equations characterizing the precise error of M-estimators

Improved Minimum Entropy Filtering for Continuous Nonlinear Non-Gaussian Systems Using a Generalized Density Evolution Equation

Adaptive filtering by minimizing error's entropy power

Optimality in Mean Estimation: Beyond Worst-Case, Beyond Sub-Gaussian, and Beyond $1+α$ Moments

Optimal estimator for uncertainty-based measurement quality control