Abstract:Gaussian process (GP) regression is a fundamental tool in Bayesian statistics. It is also known as kriging and is the Bayesian counterpart to the frequentist kernel ridge regression. Most of the theoretical work on GP regression has focused on a large-$n$ asymptotics, i.e. as the amount of data increases. Fixed-sample analysis is much more difficult outside of simple cases, such as locations on a regular grid. In this work we perform a fixed-sample analysis that was first studied in the context of approximation theory by Driscoll & Fornberg (2002), called the ``flat limit''. In flat-limit asymptotics, the goal is to characterise kernel methods as the length-scale of the kernel function tends to infinity, so that kernels appear flat over the range of the data. Surprisingly, this limit is well-defined, and displays interesting behaviour: Driscoll & Fornberg showed that radial basis interpolation converges in the flat limit to polynomial interpolation, if the kernel is Gaussian. Subsequent work showed that this holds true in the multivariate setting as well, but that kernels other than the Gaussian may have (polyharmonic) splines as the limit interpolant. Leveraging recent results on the spectral behaviour of kernel matrices in the flat limit, we study the flat limit of Gaussian process regression. Results show that Gaussian process regression tends in the flat limit to (multivariate) polynomial regression, or (polyharmonic) spline regression, depending on the kernel. Importantly, this holds for both the predictive mean and the predictive variance, so that the posterior predictive distributions become equivalent. Our results have practical consequences: for instance, they show that optimal GP predictions in the sense of leave-one-out loss may occur at very large length-scales, which would be invisible to current implementations because of numerical difficulties.

Rectangularization of Gaussian process regression for optimization of hyperparameters

Gaussian Process Regression for Geometry Optimization

Easy representation of multivariate functions with low-dimensional terms via Gaussian process regression kernel design: applications to machine learning of potential energy surfaces and kinetic energy densities from sparse data

Standard Gaussian Process Can Be Excellent for High-Dimensional Bayesian Optimization

Global Optimization of Gaussian processes

A Solution to the Ill-Conditioning of Gradient-Enhanced Covariance Matrices for Gaussian Processes

Efficient Two-Stage Gaussian Process Regression Via Automatic Kernel Search and Subsampling

Scaling Gaussian Process Regression with Derivatives

A Unified Gaussian Process for Branching and Nested Hyperparameter Optimization

Quantum Gaussian Process Regression for Bayesian Optimization

Using Gaussian process regression for efficient parameter reconstruction

H-GPR: A HYBRID STRATEGY FOR LARGE-SCALE GAUSSIAN PROCESS REGRESSION

Additive Gaussian process regression for metamodeling in high-dimensional problems

Gaussian Processes and Bayesian Optimization for High Precision Experiments

High-dimensional mixed-categorical Gaussian processes with application to multidisciplinary design optimization for a green aircraft

Towards Scalable Gaussian Process Modeling

Zero-Order Optimization for Gaussian Process-based Model Predictive Control

Physics Inspired Approaches To Understanding Gaussian Processes

Gaussian Process Regression in the Flat Limit

The loss of the property of locality of the kernel in high-dimensional Gaussian process regression on the example of the fitting of molecular potential energy surfaces

Optimal Designs of Gaussian Processes with Budgets for Hyperparameter Optimization