Abstract:Gaussian processes (GPs) can provide a principled approach to uncertainty quantification with easy-to-interpret kernel hyperparameters, such as the lengthscale, which controls the correlation distance of function values. However, selecting an appropriate kernel can be challenging. Deep GPs avoid manual kernel engineering by successively parameterizing kernels with GP layers, allowing them to learn low-dimensional embeddings of the inputs that explain the output data. Following the architecture of deep neural networks, the most common deep GPs warp the input space layer-by-layer but lose all the interpretability of shallow GPs. An alternative construction is to successively parameterize the lengthscale of a kernel, improving the interpretability but ultimately giving away the notion of learning lower-dimensional embeddings. Unfortunately, both methods are susceptible to particular pathologies which may hinder fitting and limit their interpretability. This work proposes a novel synthesis of both previous approaches: Thin and Deep GP (TDGP). Each TDGP layer defines locally linear transformations of the original input data maintaining the concept of latent embeddings while also retaining the interpretation of lengthscales of a kernel. Moreover, unlike the prior solutions, TDGP induces non-pathological manifolds that admit learning lower-dimensional representations. We show with theoretical and experimental results that i) TDGP is, unlike previous models, tailored to specifically discover lower-dimensional manifolds in the input data, ii) TDGP behaves well when increasing the number of layers, and iii) TDGP performs well in standard benchmark datasets.

Linear Time GPs for Inferring Latent Trajectories from Neural Spike Trains

Infinite-Horizon Gaussian Processes

A Fast GP Regression Method Using Banded Sparsification of Inverse Covariance

Spatio-Temporal GP Model Learning for Intention-Driven Motions.

Latent Gaussian Process Regression

Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structure

Modeling Latent Neural Dynamics with Gaussian Process Switching Linear Dynamical Systems

Implicit Gaussian process representation of vector fields over arbitrary latent manifolds

Temporal Gaussian Process Regression in Logarithmic Time

Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models

Thin and Deep Gaussian Processes

Time-Varying Transition Matrices with Multi-task Gaussian Processes

Markovian Gaussian Process: A Universal State-Space Representation for Stationary Temporal Gaussian Process

Variational Inference for Uncertainty on the Inputs of Gaussian Process Models

Amortized Variational Inference for Deep Gaussian Processes

Neural Dynamics Discovery via Gaussian Process Recurrent Neural Networks

Towards Efficient Modeling and Inference in Multi-Dimensional Gaussian Process State-Space Models

Scalable Multi-Task Gaussian Processes with Neural Embedding of Coregionalization

Large Linear Multi-output Gaussian Process Learning

Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo

Deep Neural Networks as Gaussian Processes