Abstract:The advent of large-scale inference has spurred reexamination of conventional statistical thinking. In a Gaussian model for $n$ many $z$-scores with at most $k < \frac{n}{2}$ nonnulls, Efron suggests estimating the location and scale parameters of the null distribution. Placing no assumptions on the nonnull effects, the statistical task can be viewed as a robust estimation problem. However, the best known robust estimators fail to be consistent in the regime $k \asymp n$ which is especially relevant in large-scale inference. The failure of estimators which are minimax rate-optimal with respect to other formulations of robustness (e.g. Huber's contamination model) might suggest the impossibility of consistent estimation in this regime and, consequently, a major weakness of Efron's suggestion. A sound evaluation of Efron's model thus requires a complete understanding of consistency. We sharply characterize the regime of $k$ for which consistent estimation is possible and further establish the minimax estimation rates. It is shown consistent estimation of the location parameter is possible if and only if $\frac{n}{2} - k = \omega(\sqrt{n})$, and consistent estimation of the scale parameter is possible in the entire regime $k < \frac{n}{2}$. Faster rates than those in Huber's contamination model are achievable by exploiting the Gaussian character of the data. The minimax upper bound is obtained by considering estimators based on the empirical characteristic function. The minimax lower bound involves constructing two marginal distributions whose characteristic functions match on a wide interval containing zero. The construction notably differs from those in the literature by sharply capturing a scaling of $n-2k$ in the minimax estimation rate of the location.

Efficient Estimates and Optimum Inference Procedures in Large Samples

Enhanced Estimation of Population Mean in the Presence of Auxiliary Information

Efficient large deviation estimation based on importance sampling

A Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations

Statistical inference and nonparametric efficiency: A selective survey

On Statistical Efficiency in Learning

Efficient Randomized-Adaptive Designs

The radius of statistical efficiency

Efficient combination of observational and experimental datasets under general restrictions on outcome mean functions

Optimal estimation of the null distribution in large-scale inference

Large-sample theory for inferential models: a possibilistic Bernstein--von Mises theorem

The estimation error of general first order methods

Instance-Optimality in I/O-Efficient Sampling and Sequential Estimation

Semiparametric Efficient Inference in Adaptive Experiments

Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selection

Optimal rates of convergence for estimating the null density and proportion of nonnull effects in large-scale multiple testing

Efficiency of reversible MCMC methods: elementary derivations and applications to composite methods

Efficient estimation for ergodic diffusion processes sampled at high frequency

Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy

Optimal inference in a class of regression models

Optimal distributions for randomized unbiased estimators with an infinite horizon and an adaptive algorithm