Abstract:Bayesian optimization (BO) primarily uses Gaussian processes (GP) as the key surrogate model, mostly with a simple stationary and separable kernel function such as the squared-exponential kernel with automatic relevance determination (SE-ARD). However, such simple kernel specifications are deficient in learning functions with complex features, such as being nonstationary, nonseparable, and multimodal. Approximating such functions using a local GP, even in a low-dimensional space, requires a large number of samples, not to mention in a high-dimensional setting. In this paper, we propose to use Bayesian Kernelized Tensor Factorization (BKTF) -- as a new surrogate model -- for BO in a $D$-dimensional Cartesian product space. Our key idea is to approximate the underlying $D$-dimensional solid with a fully Bayesian low-rank tensor CP decomposition, in which we place GP priors on the latent basis functions for each dimension to encode local consistency and smoothness. With this formulation, information from each sample can be shared not only with neighbors but also across dimensions. Although BKTF no longer has an analytical posterior, we can still efficiently approximate the posterior distribution through Markov chain Monte Carlo (MCMC) and obtain prediction and full uncertainty quantification (UQ). We conduct numerical experiments on both standard BO test functions and machine learning hyperparameter tuning problems, and our results show that BKTF offers a flexible and highly effective approach for characterizing complex functions with UQ, especially in cases where the initial sample size and budget are severely limited.

A Bayesian Approach for Simultaneously Radial Kernel Parameter Tuning in the Partition of Unity Method

Parameter tuning in the radial kernel-based partition of unity method by Bayesian optimization

Bayesian Approach for Radial Kernel Parameter Tuning

Partition of unity interpolation using stable kernel-based techniques

On the search of the shape parameter in radial basis functions using univariate global optimization methods

Learning a robust shape parameter for RBF approximation

Rational RBF-based partition of unity method for efficiently and accurately approximating 3D objects

A Simultaneous Strategy Based on Radial Basis Function for Dynamic Optimization

Adaptive selection strategy of shape parameters for LRBF for solving partial differential equations

Optimal design for kernel interpolation: Applications to uncertainty quantification

A Partition of Unity Method for Divergence-free or Curl-free Radial Basis Function Approximation

Local Error Estimates for Radial Basis Function Interpolation of Scattered Data

Stabilized interpolation using radial basis functions augmented with selected radial polynomials

Robust radial basis function interpolation based on geodesic distance for the numerical coupling of multiphysics problems

A study on radial basis function and quasi-Monte Carlo methods

Interpolation and Best Approximation for Spherical Radial Basis Function Networks

On the Factors Affecting the Accuracy and Robustness of Smoothed-Radial Point Interpolation Method

Multivariate Quasi-Interpolation in L-P(R-D) with Radial Basis Functions for Scattered Data

A Discrete Adapted Hierarchical Basis Solver For Radial Basis Function Interpolation

Bayesian Kernelized Tensor Factorization as Surrogate for Bayesian Optimization