Abstract:Gaussian process regression (GPR) is a powerful machine learning method which has recently enjoyed wider use, in particular in physical sciences. In its original formulation, GPR uses a square matrix of covariances among training data and can be viewed as linear regression problem with equal numbers of training data and basis functions. When data are sparse, avoidance of overfitting and optimization of hyperparameters of GPR are difficult, in particular in high-dimensional spaces where the data sparsity issue cannot practically be resolved by adding more data. Optimal choice of hyperparameters, however, determines success or failure of the application of the GPR method. We show that parameter optimization is facilitated by rectangularization of the defining equation of GPR. On the example of a 15-dimensional molecular potential energy surface we demonstrate that this approach allows effective hyperparameter tuning even with very sparse data.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the hyper - parameter optimization problem of Gaussian process regression (GPR) in high - dimensional space, especially in the case of sparse data. Specifically: 1. **Over - fitting problem caused by sparse data**: In high - dimensional space, data points are usually sparse, which makes it difficult to avoid over - fitting and optimize the hyper - parameters of GPR. 2. **Limitations of traditional methods**: Traditional methods such as maximum likelihood estimation (MLE) are difficult to find suitable hyper - parameters when data is sparse, thus affecting the accuracy and generalization ability of the model. 3. **Challenges in hyper - parameter optimization**: Selecting the optimal hyper - parameters is crucial for the successful application of GPR, but in high - dimensional space, due to sparse data, it is difficult to solve the problem by increasing the amount of data. To solve these problems, the author proposes a new method, that is, to optimize the hyper - parameters by "rectangularizing" the defining equation of GPR. The core idea of this method is to regard GPR as a regularized linear regression problem and perform hyper - parameter optimization by solving a rectangular matrix equation. The specific steps are as follows: - **Introducing the rectangular matrix equation**: By constructing an \(N\times M\) rectangular matrix (\(N > M\)), the least - squares method can be used to solve the hyper - parameters instead of relying on the traditional \(M\times M\) square matrix. - **Optimizing hyper - parameters**: By minimizing the residual of the rectangular matrix equation, the hyper - parameters can be effectively optimized, and good results can be achieved even in the case of sparse data. - **Reducing computational cost**: Compared with traditional GPR methods, the rectangularization method can use fewer basis functions, thereby reducing computational cost. Through these improvements, the author shows the effectiveness of this method in dealing with high - dimensional molecular potential energy surfaces (such as the 15 - dimensional potential energy surface of UF6 molecules), and can achieve effective hyper - parameter tuning under very sparse data conditions.

Rectangularization of Gaussian process regression for optimization of hyperparameters

Easy representation of multivariate functions with low-dimensional terms via Gaussian process regression kernel design: applications to machine learning of potential energy surfaces and kinetic energy densities from sparse data

Standard Gaussian Process Can Be Excellent for High-Dimensional Bayesian Optimization

Global Optimization of Gaussian processes

A Solution to the Ill-Conditioning of Gradient-Enhanced Covariance Matrices for Gaussian Processes

Efficient Two-Stage Gaussian Process Regression Via Automatic Kernel Search and Subsampling

Scaling Gaussian Process Regression with Derivatives

A Unified Gaussian Process for Branching and Nested Hyperparameter Optimization

Quantum Gaussian Process Regression for Bayesian Optimization

Using Gaussian process regression for efficient parameter reconstruction

H-GPR: A HYBRID STRATEGY FOR LARGE-SCALE GAUSSIAN PROCESS REGRESSION

Gaussian Processes and Bayesian Optimization for High Precision Experiments

High-dimensional mixed-categorical Gaussian processes with application to multidisciplinary design optimization for a green aircraft

Towards Scalable Gaussian Process Modeling

Zero-Order Optimization for Gaussian Process-based Model Predictive Control

Physics Inspired Approaches To Understanding Gaussian Processes

Gaussian Process Regression in the Flat Limit

The loss of the property of locality of the kernel in high-dimensional Gaussian process regression on the example of the fitting of molecular potential energy surfaces

Optimal Designs of Gaussian Processes with Budgets for Hyperparameter Optimization

Bias-Free Scalable Gaussian Processes via Randomized Truncations

Robust and Conjugate Gaussian Process Regression