Abstract:Gaussian processes provide probabilistic surrogates for various applications including classification, uncertainty quantification, and optimization. Using a gradient-enhanced covariance matrix can be beneficial since it provides a more accurate surrogate relative to its gradient-free counterpart. An acute problem for Gaussian processes, particularly those that use gradients, is the ill-conditioning of their covariance matrices. Several methods have been developed to address this problem for gradient-enhanced Gaussian processes but they have various drawbacks such as limiting the data that can be used, imposing a minimum distance between evaluation points in the parameter space, or constraining the hyperparameters. In this paper a new method is presented that applies a diagonal preconditioner to the covariance matrix along with a modest nugget to ensure that the condition number of the covariance matrix is bounded, while avoiding the drawbacks listed above. Optimization results for a gradient-enhanced Bayesian optimizer with the Gaussian kernel are compared with the use of the new method, a baseline method that constrains the hyperparameters, and a rescaling method that increases the distance between evaluation points. The Bayesian optimizer with the new method converges the optimality, ie the $\ell_2$ norm of the gradient, an additional 5 to 9 orders of magnitude relative to when the baseline method is used and it does so in fewer iterations than with the rescaling method. The new method is available in the open source python library GpGradPy, which can be found at <a class="link-external link-https" href="https://github.com/marchildon/gpgradpy/tree/paper_precon" rel="external noopener nofollow">this https URL</a>. All of the figures in this paper can be reproduced with this library.

Hyperparameter Selection for Gaussian Process One-Class Classification.

Parameter selection of Gaussian kernel for one-class SVM.

Gaussian process and its applications in hyperspectral image classification

A spatial Gaussian process method for hyperspectral remote sensing imagery classification

Variational Mixtures of Gaussian Processes for Classification.

Gaussian Process Approach to Remote Sensing Image Classification

Band selection based gaussian processes for hyperspectral remote sensing images classification

Two Methods of Selecting Gaussian Kernel Parameters for One-Class Svm and Their Application to Fault Detection

A Novel Hyperspectral Remote Sensing Images Classification Using Gaussian Processes with Conditional Random Fields

A Solution to the Ill-Conditioning of Gradient-Enhanced Covariance Matrices for Gaussian Processes

Speeding up the binary Gaussian process classification

A Unified Gaussian Process for Branching and Nested Hyperparameter Optimization

An Approach to Choosing Gaussian Kernel Parameter for One-Class SVMs Via Tightness Detecting

Penalized gaussian process regression and classification for high-dimensional nonlinear data

Epoch-Evolving Gaussian Process Guided Learning for Classification

Provably Efficient Bayesian Optimization with Unbiased Gaussian Process Hyperparameter Estimation

Parallel cross-validation: A scalable fitting method for Gaussian process models

Variable Sigma Gaussian Processes: an Expectation Propagation Perspective

Differential Privacy With Variant-Noise For Gaussian Processes Classification

How Priors of Initial Hyperparameters Affect Gaussian Process Regression Models

Nonstationary Gaussian Process Discriminant Analysis With Variable Selection for High-Dimensional Functional Data