Abstract:We study the posterior contraction rates of a Bayesian method with Gaussian process priors in nonparametric regression and its plug-in property for differential operators. For a general class of kernels, we establish convergence rates of the posterior measure of the regression function and its derivatives, which are both minimax optimal up to a logarithmic factor for functions in certain classes. Our calculation shows that the rate-optimal estimation of the regression function and its derivatives share the same choice of hyperparameter, indicating that the Bayes procedure remarkably adapts to the order of derivatives and enjoys a generalized plug-in property that extends real-valued functionals to function-valued functionals. This leads to a practically simple method for estimating the regression function and its derivatives, whose finite sample performance is assessed using simulations. Our proof shows that, under certain conditions, to any convergence rate of Bayes estimators there corresponds the same convergence rate of the posterior distributions (i.e., posterior contraction rate), and vice versa. This equivalence holds for a general class of Gaussian processes and covers the regression function and its derivative functionals, under both the $L_2$ and $L_{\infty}$ norms. In addition to connecting these two fundamental large sample properties in Bayesian and non-Bayesian regimes, such equivalence enables a new routine to establish posterior contraction rates by calculating convergence rates of nonparametric point estimators. At the core of our argument is an operator-theoretic framework for kernel ridge regression and equivalent kernel techniques. We derive a range of sharp non-asymptotic bounds that are pivotal in establishing convergence rates of nonparametric point estimators and the equivalence theory, which may be of independent interest.

A Characterization of Conjugate Priors in Exponential Families with Application to Inverse Regression

Posteriors, conjugacy, and exponential families for completely random measures

Conjugate Analysis Of The Conway-Maxwell-Poisson Distribution (Vol 1, Pg 363, 2006)

A Geometric View of Conjugate Priors

A Channel-Based Perspective on Conjugate Priors

The conjugate prior for discrete hierarchical log-linear models

Estimation in Functional Regression for General Exponential Families

Equivalence of Convergence Rates of Posterior Distributions and Bayes Estimators for Functions and Nonparametric Functionals

On a method to construct exponential families by representation theory

Horseshoe Priors for Sparse Dirichlet-Multinomial Models

A Unified Theory of Exact Inference and Learning in Exponential Family Latent Variable Models

Note of Correction: “conjugate Analysis of the Conway-Maxwell-Poisson Distribution”

Compound Poisson Processes, Latent Shrinkage Priors and Bayesian Nonconvex Penalization

Convergence of Pseudo-Bayes Factors in Forward and Inverse Regression Problems

EP-GIG Priors and Applications in Bayesian Sparse Learning

Conjugacy properties of multivariate unified skew-elliptical distributions

On Non- and Weakly-Informative Priors for the Conway-Maxwell-Poisson (COM-Poisson) Distribution

Enriched standard conjugate priors and the right invariant prior for Wishart distributions

High-dimensional Posterior Consistency in Multi-response Regression models with Non-informative Priors for Error Covariance Matrix

Bayesian Nonparametric Covariance Regression