Abstract:The Gaussian process (GP) is a widely used probabilistic machine learning method for stochastic function approximation, stochastic modeling, and analyzing real-world measurements of nonlinear processes. Unlike many other machine learning methods, GPs include an implicit characterization of uncertainty, making them extremely useful across many areas of science, technology, and engineering. Traditional implementations of GPs involve stationary kernels (also termed covariance functions) that limit their flexibility and exact methods for inference that prevent application to data sets with more than about ten thousand points. Modern approaches to address stationarity assumptions generally fail to accommodate large data sets, while all attempts to address scalability focus on approximating the Gaussian likelihood, which can involve subjectivity and lead to inaccuracies. In this work, we explicitly derive an alternative kernel that can discover and encode both sparsity and nonstationarity. We embed the kernel within a fully Bayesian GP model and leverage high-performance computing resources to enable the analysis of massive data sets. We demonstrate the favorable performance of our novel kernel relative to existing exact and approximate GP methods across a variety of synthetic data examples. Furthermore, we conduct space-time prediction based on more than one million measurements of daily maximum temperature and verify that our results outperform state-of-the-art methods in the Earth sciences. More broadly, having access to exact GPs that use ultra-scalable, sparsity-discovering, nonstationary kernels allows GP methods to truly compete with a wide variety of machine learning methods.

A General Theory for Kernel Packets: from state space model to compactly supported basis

Kernel Generated Two-Time Parameter Gaussian Processes and Some of Their Path Properties

Active data based Gaussian process models for nonlinear spatiotemporal systems

Kernel Multigrid: Accelerate Back-fitting via Sparse Gaussian Process Regression

Compactly-supported nonstationary kernels for computing exact Gaussian processes on big data

A Unifying Perspective on Non-Stationary Kernels for Deeper Gaussian Processes

Compressing spectral kernels in Gaussian Process: Enhanced generalization and interpretability

Representing Additive Gaussian Processes by Sparse Matrices

Additive Kernels for Gaussian Process Modeling

A General Kernelization Framework for Learning Algorithms Based on Kernel PCA.

Compressible Spectral Mixture Kernels with Sparse Dependency Structures for Gaussian Processes

EigenGP: Gaussian Process Models with Adaptive Eigenfunctions.

On the Identifiability and Interpretability of Gaussian Process Models

Further Understanding of a Local Gaussian Process Approximation: Characterising Convergence in the Finite Regime

Linear Time GPs for Inferring Latent Trajectories from Neural Spike Trains

Scalable marginalization of correlated latent variables with applications to learning particle interaction kernels

Tensor Network-Constrained Kernel Machines as Gaussian Processes

State Space representation of non-stationary Gaussian Processes

EigenGP: Gaussian Processes with Sparse Data-Dependent Eigenfunctions

Bayesian Generalized Kernel Mixed Models