Abstract:Empirical Bayes (EB) is a popular framework for large-scale inference that aims to find data-driven estimators to compete with the Bayesian oracle that knows the true prior. Two principled approaches to EB estimation have emerged over the years: $f$-modeling, which constructs an approximate Bayes rule by estimating the marginal distribution of the data, and $g$-modeling, which estimates the prior from data and then applies the learned Bayes rule. For the Poisson model, the prototypical examples are the celebrated Robbins estimator and the nonparametric MLE (NPMLE), respectively. It has long been recognized in practice that the Robbins estimator, while being conceptually appealing and computationally simple, lacks robustness and can be easily derailed by ``outliers'', unlike the NPMLE which provides more stable and interpretable fit thanks to its Bayes form. On the other hand, not only do the existing theories shed little light on this phenomenon, but they all point to the opposite, as both methods have recently been shown optimal in terms of regret (excess over the Bayes risk) for compactly supported and subexponential priors. In this paper we provide a theoretical justification for the superiority of $g$-modeling over $f$-modeling for heavy-tailed data by considering priors with bounded $p>1$th moment. We show that with mild regularization, any $g$-modeling method that is Hellinger rate-optimal in density estimation achieves an optimal total regret $\tilde \Theta(n^{\frac{3}{2p+1}})$; in particular, the special case of NPMLE succeeds without regularization. In contrast, there exists an $f$-modeling estimator whose density estimation rate is optimal but whose EB regret is suboptimal by a polynomial factor. These results show that the proper Bayes form provides a ``general recipe of success'' for optimal EB estimation that applies to all $g$-modeling (but not $f$-modeling) methods.

The Superiorities of Empirical Bayes Estimation of Variance Components in Random Effects Model

Empirical Bayes Estimators of Risk Premium under Variance Related Premium Principle

A Comparison of REML and Bayesian Estimation of Variance Components for Unbalanced One-way Random Effects Models With Small Sample Sizes

The Superiority of Bayes Linear Unbiased Minimum Variance Estimator with Respect to Ridge Estimator

The Superiority of the Bayes Linear Unbiased Minimum Variance Estimator under Balanced Loss Function

Purification of human serum amyloid A by anion-exchange fast protein liquid chromatography.

Empirical Bayes When Estimation Precision Predicts Parameters

On estimation of mean squared errors of benchmarked empirical Bayes estimators

Estimation in Covariance Components Models

Bias correction for multiple covariate analysis using empirical bayesian estimation in mixed-effects models for longitudinal data

Empirical Bayes in Bayesian learning: understanding a common practice

An Evaluation of Empirical Bayes’s Estimation of Value-Added Teacher Performance Measures

Empirical Bayes Estimates for a 2-Way Cross-Classified Additive Model

Entropy Balancing is Doubly Robust

Empirical Bayes for Large-scale Randomized Experiments: a Spectral Approach

Empirical Bayes via ERM and Rademacher complexities: the Poisson model

Nonparametric estimation of multivariate copula using empirical bayes method

Empirical Bayes estimation: When does $g$-modeling beat $f$-modeling in theory (and in practice)?

Learning from a lot: Empirical Bayes in high-dimensional prediction settings

Sparse high-dimensional linear regression with a partitioned empirical Bayes ECM algorithm

Bayesian Repeated-Measures Analysis of Variance: An Updated Methodology Implemented in JASP