Abstract:It is commonly believed that Bayesian optimization (BO) algorithms are highly efficient for optimizing numerically costly functions. However, BO is not often compared to widely different alternatives, and is mostly tested on narrow sets of problems (multimodal, low-dimensional functions), which makes it difficult to assess where (or if) they actually achieve state-of-the-art performance. Moreover, several aspects in the design of these algorithms vary across implementations without a clear recommendation emerging from current practices, and many of these design choices are not substantiated by authoritative test campaigns. This article reports a large investigation about the effects on the performance of (Gaussian process based) BO of common and less common design choices. The following features are considered: the size of the initial design of experiments, the functional form of the trend, the choice of the kernel, the internal optimization strategy, input or output warping, and the use of the Gaussian process (GP) mean in conjunction with the classical Expected Improvement. The experiments are carried out with the established COCO (COmparing Continuous Optimizers) software. It is found that a small initial budget, a quadratic trend, high-quality optimization of the acquisition criterion bring consistent progress. Using the GP mean as an occasional acquisition contributes to a negligible additional improvement. Warping degrades performance. The Matérn 5/2 kernel is a good default but it may be surpassed by the exponential kernel on irregular functions. Overall, the best EGO variants are competitive or improve over state-of-the-art algorithms in dimensions less or equal to 5 for multimodal functions. The code developed for this study makes the new version (v2.1.1) of the R package DiceOptim available on CRAN. The structure of the experiments by function groups allows to define priorities for future research on Bayesian optimization.

Bayesian Optimization Based on K-Optimality

Process Quality Control Through Bayesian Optimization with Adaptive Local Convergence

How Bayesian Should Bayesian Optimisation Be?

Robust Bayesian Optimization via Localized Online Conformal Prediction

Poisson Process for Bayesian Optimization

Provably Efficient Bayesian Optimization with Unbiased Gaussian Process Hyperparameter Estimation

Bayesian Optimization Using Pseudo-Points

A Sequential Sampling-Based Bayesian Numerical Method for Reliability-Based Design Optimization

Simulation Based Bayesian Optimization

Deep Kernel Learning-Based Bayesian Optimization with Adaptive Kernel Functions

Approximation-Aware Bayesian Optimization

Modulating Surrogates for Bayesian Optimization

Co-Learning Bayesian Optimization.

Revisiting Bayesian optimization in the light of the COCO benchmark

A Bayesian Optimization Framework for Finding Local Optima in Expensive Multimodal Functions

A Bayesian Optimization Framework for Finding Local Optima in Expensive Multi-Modal Functions

Pseudo-Bayesian Optimization

Bayesian Optimization under Stochastic Delayed Feedback

BOIS: Bayesian Optimization of Interconnected Systems

Extrapolative Bayesian Optimization with Gaussian Process and Neural Network Ensemble Surrogate Models

Bayesian optimization as a flexible and efficient design framework for sustainable process systems