Abstract:We propose practical deep Gaussian process models on Riemannian manifolds, similar in spirit to residual neural networks. With manifold-to-manifold hidden layers and an arbitrary last layer, they can model manifold- and scalar-valued functions, as well as vector fields. We target data inherently supported on manifolds, which is too complex for shallow Gaussian processes thereon. For example, while the latter perform well on high-altitude wind data, they struggle with the more intricate, nonstationary patterns at low altitudes. Our models significantly improve performance in these settings, enhancing prediction quality and uncertainty calibration, and remain robust to overfitting, reverting to shallow models when additional complexity is unneeded. We further showcase our models on Bayesian optimisation problems on manifolds, using stylised examples motivated by robotics, and obtain substantial improvements in later stages of the optimisation process. Finally, we show our models to have potential for speeding up inference for non-manifold data, when, and if, it can be mapped to a proxy manifold well enough.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to construct a deep Gaussian process model on manifolds that can handle complex and irregular data. Traditional shallow Gaussian processes perform poorly when dealing with complex data on manifolds, especially when the data has complex non - stationary patterns, such as wind - speed data at low altitudes. To overcome these limitations, the authors propose Residual Deep Gaussian Processes (Residual Deep GPs), which is a deep Gaussian process model constructed on Riemannian manifolds. By introducing residual connections (similar to the concept in residual neural networks), the model can better capture complex data patterns and improve the accuracy of prediction quality and uncertainty calibration. In addition, this model can also degenerate into a shallow model without additional complexity, thereby avoiding over - fitting. Specifically, the main contributions of this paper include: 1. **Model Architecture**: A new method for constructing deep Gaussian processes on Riemannian manifolds is proposed, which is achieved by using Gaussian Vector Fields (GVFs) and exponential mapping to realize the transformation from manifold to manifold. 2. **Variational Inference**: A doubly stochastic variational inference method suitable for Residual Deep Gaussian Processes has been developed, including the use of inducing variables and cross - domain inducing variables to accelerate the inference process. 3. **Experimental Verification**: Through experiments on synthetic data and real - world data, the superior performance of this model in regression tasks, Bayesian optimization, and wind - speed prediction tasks has been demonstrated, especially when dealing with complex and irregular data. In summary, this paper aims to solve the deficiencies of existing methods in handling complex data on manifolds by proposing the Residual Deep Gaussian Processes model, thereby improving the prediction ability and robustness of the model.

Residual Deep Gaussian Processes on Manifolds

Residual Gaussian Process: A Tractable Nonparametric Bayesian Emulator for Multi-Fidelity Simulations

Manifold Gaussian Processes for Regression

Deep Manifold Computing and Visualization

Intrinsic Gaussian Process on Unknown Manifolds with Probabilistic Metrics

Intrinsic Gaussian Vector Fields on Manifolds

Implicit Gaussian process representation of vector fields over arbitrary latent manifolds

Dynamically Stable Poincaré Embeddings for Neural Manifolds

Implicit Manifold Gaussian Process Regression

Intrinsic Wrapped Gaussian Process Regression Modeling for Manifold-valued Response Variable

Thin and Deep Gaussian Processes

Posterior Contraction Rates for Matérn Gaussian Processes on Riemannian Manifolds

Sparse Gaussian Processes with Manifold-Preserving Graph Reduction

Beyond the Mean-Field: Structured Deep Gaussian Processes Improve the Predictive Uncertainties

ManifoldNet: A Deep Network Framework for Manifold-valued Data

Riemannian Residual Neural Networks

Manifold Optimization Assisted Gaussian Variational Approximation

Manifold Learning for the Emulation of Spatial Fields from Computational Models.

Building Neural Networks on Matrix Manifolds: A Gyrovector Space Approach

High-Dimensional Intrinsic Interpolation Using Gaussian Process Regression and Diffusion Maps

Matrix Manifold Neural Networks++