Abstract:The vast quantity of information brought by big data as well as the evolving computer hardware encourages success stories in the machine learning community. In the meanwhile, it poses challenges for the Gaussian process regression (GPR), a well-known nonparametric, and interpretable Bayesian model, which suffers from cubic complexity to data size. To improve the scalability while retaining desirable prediction quality, a variety of scalable GPs have been presented. However, they have not yet been comprehensively reviewed and analyzed to be well understood by both academia and industry. The review of scalable GPs in the GP community is timely and important due to the explosion of data size. To this end, this article is devoted to reviewing state-of-the-art scalable GPs involving two main categories: global approximations that distillate the entire data and local approximations that divide the data for subspace learning. Particularly, for global approximations, we mainly focus on sparse approximations comprising prior approximations that modify the prior but perform exact inference, posterior approximations that retain exact prior but perform approximate inference, and structured sparse approximations that exploit specific structures in kernel matrix; for local approximations, we highlight the mixture/product of experts that conducts model averaging from multiple local experts to boost predictions. To present a complete review, recent advances for improving the scalability and capability of scalable GPs are reviewed. Finally, the extensions and open issues of scalable GPs in various scenarios are reviewed and discussed to inspire novel ideas for future research avenues.

Scalable Gaussian Process Regression Using Deep Neural Networks.

Deep Graph Gaussian Processes for Short-Term Traffic Flow Forecasting from Spatiotemporal Data

Asynchronous Parallel Large-Scale Gaussian Process Regression

Deep Neural Networks as Gaussian Processes

Modulating Scalable Gaussian Processes for Expressive Statistical Learning

Neural Network Gaussian Processes by Increasing Depth

Fast and Efficient DNN Deployment via Deep Gaussian Transfer Learning

Understanding and comparing scalable Gaussian process regression for big data

Scalable Training of Inference Networks for Gaussian-Process Models.

Gaussian Process Neural Additive Models

Deep Factors with Gaussian Processes for Forecasting

When Gaussian Process Meets Big Data: A Review of Scalable GPs

Hierarchical Mixture-of-Experts Model for Large-Scale Gaussian Process Regression

Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inference

Reinforcement Learning via Gaussian Processes with Neural Network Dual Kernels

Node-level Graph Regression with Deep Gaussian Process Models

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

Efficient Learning Algorithms for Gaussian Processes

Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models

Scaling Gaussian Process Regression with Derivatives

Asynchronous Distributed Variational Gaussian Process for Regression.