Abstract:It is commonly believed that Bayesian optimization (BO) algorithms are highly efficient for optimizing numerically costly functions. However, BO is not often compared to widely different alternatives, and is mostly tested on narrow sets of problems (multimodal, low-dimensional functions), which makes it difficult to assess where (or if) they actually achieve state-of-the-art performance. Moreover, several aspects in the design of these algorithms vary across implementations without a clear recommendation emerging from current practices, and many of these design choices are not substantiated by authoritative test campaigns. This article reports a large investigation about the effects on the performance of (Gaussian process based) BO of common and less common design choices. The following features are considered: the size of the initial design of experiments, the functional form of the trend, the choice of the kernel, the internal optimization strategy, input or output warping, and the use of the Gaussian process (GP) mean in conjunction with the classical Expected Improvement. The experiments are carried out with the established COCO (COmparing Continuous Optimizers) software. It is found that a small initial budget, a quadratic trend, high-quality optimization of the acquisition criterion bring consistent progress. Using the GP mean as an occasional acquisition contributes to a negligible additional improvement. Warping degrades performance. The Matérn 5/2 kernel is a good default but it may be surpassed by the exponential kernel on irregular functions. Overall, the best EGO variants are competitive or improve over state-of-the-art algorithms in dimensions less or equal to 5 for multimodal functions. The code developed for this study makes the new version (v2.1.1) of the R package DiceOptim available on CRAN. The structure of the experiments by function groups allows to define priorities for future research on Bayesian optimization.

Biobjective Performance Assessment with the COCO Platform

COCO: a platform for comparing continuous optimizers in a black-box setting

Using Well-Understood Single-Objective Functions in Multiobjective Black-Box Optimization Test Suites

COCO: Performance Assessment

COCO: The Large Scale Black-Box Optimization Benchmarking (bbob-largescale) Test Suite

MA-BBOB: Many-Affine Combinations of BBOB Functions for Evaluating AutoML Approaches in Noiseless Numerical Black-Box Optimization Contexts

Anytime Bi-Objective Optimization with a Hybrid Multi-Objective CMA-ES (HMO-CMA-ES)

COCO: The Experimental Procedure

Using Affine Combinations of BBOB Problems for Performance Assessment

A Complementarity Analysis of the COCO Benchmark Problems and Artificially Generated Problems

An efficient chemical reaction algorithm for multi-objective combinatorial bi-level optimization

Bonobo optimizer (BO): an intelligent heuristic with self-adjusting parameters over continuous spaces and its applications to engineering problems

Revisiting Bayesian optimization in the light of the COCO benchmark

Benchmarking Real-World Many-Objective Problems: A Problem Suite With Baseline Results

Comparison of High-Dimensional Bayesian Optimization Algorithms on BBOB

Framework and Benchmarks for Combinatorial and Mixed-variable Bayesian Optimization

Handling Constrained Multiobjective Optimization Problems via Bidirectional Coevolution

Characterization of Constrained Continuous Multiobjective Optimization Problems: A Performance Space Perspective

On Goal-Oriented Multiobjective Embedded Optimization for System Performance Assessment

Diagnostic benchmarking of many-objective evolutionary algorithms for real-world problems

Evolutionary Multitasking for Multiobjective Continuous Optimization: Benchmark Problems, Performance Metrics and Baseline Results