Abstract:There is growing interest in improving our algorithmic understanding of fundamental statistical problems such as mean estimation, driven by the goal of understanding the limits of what we can extract from valuable data. The state of the art results for mean estimation in $\mathbb{R}$ are 1) the optimal sub-Gaussian mean estimator by [LV22], with the tight sub-Gaussian constant for all distributions with finite but unknown variance, and 2) the analysis of the median-of-means algorithm by [BCL13] and a lower bound by [DLLO16], characterizing the big-O optimal errors for distributions for which only a $1+\alpha$ moment exists for $\alpha \in (0,1)$. Both results, however, are optimal only in the worst case. We initiate the fine-grained study of the mean estimation problem: Can algorithms leverage useful features of the input distribution to beat the sub-Gaussian rate, without explicit knowledge of such features? We resolve this question with an unexpectedly nuanced answer: "Yes in limited regimes, but in general no". For any distribution $p$ with a finite mean, we construct a distribution $q$ whose mean is well-separated from $p$'s, yet $p$ and $q$ are not distinguishable with high probability, and $q$ further preserves $p$'s moments up to constants. The main consequence is that no reasonable estimator can asymptotically achieve better than the sub-Gaussian error rate for any distribution, matching the worst-case result of [LV22]. More generally, we introduce a new definitional framework to analyze the fine-grained optimality of algorithms, which we call "neighborhood optimality", interpolating between the unattainably strong "instance optimality" and the trivially weak "admissibility" definitions. Applying the new framework, we show that median-of-means is neighborhood optimal, up to constant factors. It is open to find a neighborhood-optimal estimator without constant factor slackness.

Lower bounds for the trade-off between bias and mean absolute deviation

On lower bounds for the bias-variance trade-off

Optimal heteroskedasticity testing in nonparametric regression

Optimal Excess Risk Bounds for Empirical Risk Minimization on $p$-Norm Linear Regression

Lower Bounds for Nonparametric Estimation of Ordinary Differential Equations

Asymptotic Theory of Least Distances Estimate in Multivariate Linear Models

The Cost of Privacy: Optimal Rates of Convergence for Parameter Estimation with Differential Privacy

Two New Families of Local Asymptotically Minimax Lower Bounds in Parameter Estimation

Optimality in Mean Estimation: Beyond Worst-Case, Beyond Sub-Gaussian, and Beyond $1+α$ Moments

On deviation probabilities in non-parametric regression with heavy-tailed noise

Query Complexity of Least Absolute Deviation Regression via Robust Uniform Convergence

Better and Simpler Lower Bounds for Differentially Private Statistical Estimation

Information Lower Bounds for Robust Mean Estimation

Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Model

High-probability minimax lower bounds

The existence of the least favorable noise

Minimax Optimality of Score-based Diffusion Models: Beyond the Density Lower Bound Assumptions

Lower and Upper Bounds on the Generalization of Stochastic Exponentially Concave Optimization.

Noisy recovery from random linear observations: Sharp minimax rates under elliptical constraints

A Few Interactions Improve Distributed Nonparametric Estimation, Optimally

On the near-optimality of betting confidence sets for bounded means