Abstract:Bayesian Optimization (BO) is a surrogate-based global optimization strategy that relies on a Gaussian Process regression (GPR) model to approximate the objective function and an acquisition function to suggest candidate points. It is well-known that BO does not scale well for high-dimensional problems because the GPR model requires substantially more data points to achieve sufficient accuracy and acquisition optimization becomes computationally expensive in high dimensions. Several recent works aim at addressing these issues, e.g., methods that implement online variable selection or conduct the search on a lower-dimensional sub-manifold of the original search space. Advancing our previous work of PCA-BO that learns a linear sub-manifold, this paper proposes a novel kernel PCA-assisted BO (KPCA-BO) algorithm, which embeds a non-linear sub-manifold in the search space and performs BO on this sub-manifold. Intuitively, constructing the GPR model on a lower-dimensional sub-manifold helps improve the modeling accuracy without requiring much more data from the objective function. Also, our approach defines the acquisition function on the lower-dimensional sub-manifold, making the acquisition optimization more manageable. We compare the performance of KPCA-BO to a vanilla BO and to PCA-BO on the multi-modal problems of the COCO/BBOB benchmark suite. Empirical results show that KPCA-BO outperforms BO in terms of convergence speed on most test problems, and this benefit becomes more significant when the dimensionality increases. For the 60D functions, KPCA-BO achieves better results than PCA-BO for many test cases. Compared to the vanilla BO, it efficiently reduces the CPU time required to train the GPR model and to optimize the acquisition function compared to the vanilla BO.

Generalisation bounds for kernel PCA through PAC‐Bayes learning

Robust Principal Component Analysis Based on Maximum Correntropy Criterion

Cross-validatory Framework for Optimal Parameter Estimation of KPCA and KPLS Models

A General Framework for the Practical Disintegration of PAC-Bayesian Bounds

An Iterative Algorithm for Robust Kernel Principal Component Analysis

A General Kernelization Framework for Learning Algorithms Based on Kernel PCA.

A PAC-Bayesian Link Between Generalisation and Flat Minima

Kernel Generalization of PPCA for Nonlinear Probabilistic Monitoring

Leveraging PAC-Bayes Theory and Gibbs Distributions for Generalization Bounds with Complexity Measures

Better-than-KL PAC-Bayes Bounds

Empirical Evaluation of Kernel PCA Approximation Methods in Classification Tasks

Invariant subspaces and PCA in nearly matrix multiplication time

Generalized probabilistic principal component analysis of correlated data

On the Estimation Performance of Generalized Power Method for Heteroscedastic Probabilistic PCA

Learning with Cross-Kernels and Ideal PCA

High Dimensional Bayesian Optimization with Kernel Principal Component Analysis

Statistical Optimality and Computational Efficiency of Nyström Kernel PCA

Maximally Correlated Principal Component Analysis

Matrix-based Kernel Principal Component Analysis for Large-Scale Data Set.

Stochastic Optimization for Kernel Pca

Tutorial on PCA and approximate PCA and approximate kernel PCA