Abstract:Variational approximations to Gaussian processes (GPs) typically use a small set of inducing points to form a low-rank approximation to the covariance matrix. In this work, we instead exploit a sparse approximation of the precision matrix. We propose variational nearest neighbor Gaussian process (VNNGP), which introduces a prior that only retains correlations within $K$ nearest-neighboring observations, thereby inducing sparse precision structure. Using the variational framework, VNNGP's objective can be factorized over both observations and inducing points, enabling stochastic optimization with a time complexity of $O(K^3)$. Hence, we can arbitrarily scale the inducing point size, even to the point of putting inducing points at every observed location. We compare VNNGP to other scalable GPs through various experiments, and demonstrate that VNNGP (1) can dramatically outperform low-rank methods, and (2) is less prone to overfitting than other nearest neighbor methods.

What problem does this paper attempt to address?

The problems that this paper attempts to solve are the computational efficiency and prediction accuracy issues encountered when applying Gaussian Processes (GPs) to large - scale spatio - temporal data sets. Specifically: 1. **Computational efficiency problem**: Traditional Gaussian processes face the problem of high computational complexity when dealing with large - scale data, usually $O(N^3)$, where $N$ is the number of observation points. This makes it difficult to handle large - scale data sets in practical applications. 2. **Prediction accuracy problem**: For data sets with low - scale characteristics (i.e., the data changes rapidly in space and time), existing low - rank approximation methods (such as SVGP) often fail to capture these rapid changes, resulting in inaccurate or "blurred" prediction results. To address these problems, the paper proposes a new method - Variational Nearest Neighbor Gaussian Process (VNNGP), which improves the traditional method by introducing sparse precision matrix approximation. Specifically: - **Sparse precision matrix approximation**: Unlike traditional low - rank approximation, VNNGP uses sparse precision matrix approximation, assuming that each observation point is only related to its $K$ nearest neighbors. This assumption enables the model to handle low - scale data more effectively and greatly reduces the computational complexity. - **Variational inference framework**: VNNGP is optimized under the variational inference framework and can be efficiently trained using methods such as stochastic gradient descent. In addition, this method can also be used in non - conjugate likelihood functions and cross - domain observation models, increasing the scope of application of the model. Through these improvements, VNNGP can not only handle large - scale data sets but also improve the prediction accuracy while maintaining computational efficiency, especially in spatio - temporal data sets. The paper verifies the effectiveness and advantages of VNNGP through multiple experiments and shows that its performance on different data sets is better than other methods.

Variational Nearest Neighbor Gaussian Process

Variational Gaussian Processes: A Functional Analysis View

Spherical Inducing Features for Orthogonally-Decoupled Gaussian Processes

Amortized Variational Inference for Deep Gaussian Processes

Sparse Orthogonal Variational Inference for Gaussian Processes

Variable Sigma Gaussian Processes: an Expectation Propagation Perspective

Neural Operator Variational Inference based on Regularized Stein Discrepancy for Deep Gaussian Processes

Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inference

Multiview Variational Sparse Gaussian Processes.

Variational Inference for Uncertainty on the Inputs of Gaussian Process Models

Scalable Training of Inference Networks for Gaussian-Process Models.

Robust and Scalable Gaussian Process Regression and Its Applications

Radial Neighbors for Provably Accurate Scalable Approximations of Gaussian Processes

Scalable Bayesian Optimization Using Vecchia Approximations of Gaussian Processes

Leveraging Locality and Robustness to Achieve Massively Scalable Gaussian Process Regression

Asynchronous Distributed Variational Gaussian Process for Regression.

Constrained Gaussian Process for Signal Integrity applications using Variational Inference

Robust Gaussian Processes via Relevance Pursuit

Further Understanding of a Local Gaussian Process Approximation: Characterising Convergence in the Finite Regime

Learning inducing points and uncertainty on molecular data by scalable variational Gaussian processes

Vecchia Gaussian Processes: Probabilistic Properties, Minimax Rates and Methodological Developments