Abstract:The vast quantity of information brought by big data as well as the evolving computer hardware encourages success stories in the machine learning community. In the meanwhile, it poses challenges for the Gaussian process regression (GPR), a well-known nonparametric, and interpretable Bayesian model, which suffers from cubic complexity to data size. To improve the scalability while retaining desirable prediction quality, a variety of scalable GPs have been presented. However, they have not yet been comprehensively reviewed and analyzed to be well understood by both academia and industry. The review of scalable GPs in the GP community is timely and important due to the explosion of data size. To this end, this article is devoted to reviewing state-of-the-art scalable GPs involving two main categories: global approximations that distillate the entire data and local approximations that divide the data for subspace learning. Particularly, for global approximations, we mainly focus on sparse approximations comprising prior approximations that modify the prior but perform exact inference, posterior approximations that retain exact prior but perform approximate inference, and structured sparse approximations that exploit specific structures in kernel matrix; for local approximations, we highlight the mixture/product of experts that conducts model averaging from multiple local experts to boost predictions. To present a complete review, recent advances for improving the scalability and capability of scalable GPs are reviewed. Finally, the extensions and open issues of scalable GPs in various scenarios are reviewed and discussed to inspire novel ideas for future research avenues.

From Gaussian Processes to the Mixture of Gaussian Processes:A Survey

Gaussian mixture density modeling and decomposition with weighted likelihood

Mixture of Gaussian Processes and its Applications

An Effective Model Selection Criterion for Mixtures of Gaussian Processes.

Attention Mechanism Based Mixture of Gaussian Processes

Promoting Active Learning with Mixtures of Gaussian Processes

A Specialized Probability Density Function for the Input of Mixture of Gaussian Processes.

An Improved Mixture Model of Gaussian Processes and Its Classification Expectation–Maximization Algorithm

Variational Mixtures of Gaussian Processes for Classification.

Mixture Gaussian Process Model with Gaussian Mixture Distribution for Big Data

A Two-Layer Mixture Model of Gaussian Process Functional Regressions and Its MCMC EM Algorithm

A unifying view for the mixture model of sparse Gaussian processes

The whole and its parts: Visualizing Gaussian mixture models

On the Identifiability and Interpretability of Gaussian Process Models

When Gaussian Process Meets Big Data: A Review of Scalable GPs

Additive Gaussian process regression for metamodeling in high-dimensional problems

Revisiting Gaussian Process Dynamical Models.

Remarks on multivariate Gaussian Process

An Mcmc Based Em Algorithm For Mixtures Of Gaussian Processes

Multiview Learning with Variational Mixtures of Gaussian Processes

Mixture of robust Gaussian processes and its hard-cut EM algorithm with variational bounding approximation