Abstract:Empirical Bayes (EB) is a popular framework for large-scale inference that aims to find data-driven estimators to compete with the Bayesian oracle that knows the true prior. Two principled approaches to EB estimation have emerged over the years: $f$-modeling, which constructs an approximate Bayes rule by estimating the marginal distribution of the data, and $g$-modeling, which estimates the prior from data and then applies the learned Bayes rule. For the Poisson model, the prototypical examples are the celebrated Robbins estimator and the nonparametric MLE (NPMLE), respectively. It has long been recognized in practice that the Robbins estimator, while being conceptually appealing and computationally simple, lacks robustness and can be easily derailed by ``outliers'', unlike the NPMLE which provides more stable and interpretable fit thanks to its Bayes form. On the other hand, not only do the existing theories shed little light on this phenomenon, but they all point to the opposite, as both methods have recently been shown optimal in terms of regret (excess over the Bayes risk) for compactly supported and subexponential priors. In this paper we provide a theoretical justification for the superiority of $g$-modeling over $f$-modeling for heavy-tailed data by considering priors with bounded $p>1$th moment. We show that with mild regularization, any $g$-modeling method that is Hellinger rate-optimal in density estimation achieves an optimal total regret $\tilde \Theta(n^{\frac{3}{2p+1}})$; in particular, the special case of NPMLE succeeds without regularization. In contrast, there exists an $f$-modeling estimator whose density estimation rate is optimal but whose EB regret is suboptimal by a polynomial factor. These results show that the proper Bayes form provides a ``general recipe of success'' for optimal EB estimation that applies to all $g$-modeling (but not $f$-modeling) methods.

The empirical Bayes estimators of the rate parameter of the gamma distribution with a conjugate gamma prior under Stein's loss function

Variance Analyzed under E-Bayesian Estimation and Hierarchical Bayesian Estimation for Diagnosing Failure Rates

Bayesian Inference for the Gamma Zero-Truncated Poisson Distribution with an Application to Real Data

Empirical Bayes in Bayesian learning: understanding a common practice

One-sided Empirical Bayes Test for Location Parameter in Gamma Distribution

Empirical Bayes via ERM and Rademacher complexities: the Poisson model

EM Estimation for the Poisson-Inverse Gamma Regression Model with Varying Dispersion: An Application to Insurance Ratemaking

Empirical Bayes estimation: When does $g$-modeling beat $f$-modeling in theory (and in practice)?

Nonparametric Bayesian volatility estimation for gamma-driven stochastic differential equations

Estimating an Inverse Gamma distribution

Empirical Bayes estimates of steady-state availability in exponential distribution model

The Inverse Gamma-Gamma Prior for Optimal Posterior Contraction and Multiple Hypothesis Testing

Empirical Bayes estimators for the reproduction parameter of Borel-Tanner distribution

Multivariate, Heteroscedastic Empirical Bayes via Nonparametric Maximum Likelihood

Poisson and Gamma Model Marginalisation and Marginal Likelihood calculation using Moment-generating Functions

Dir-SPGLM: A Bayesian semiparametric GLM with data-driven reference distribution

Empirical Bayes inference in sparse high-dimensional generalized linear models

On Parameter Estimation in Deviated Gaussian Mixture of Experts

The Impact of Loss Estimation on Gibbs Measures

A Bayesian Analysis for the Parameters of the Exponential-Logarithmic Distribution

Sparse reconstructions from few noisy data: analysis of hierarchical Bayesian models with generalized gamma hyperpriors