Abstract:Bayesian optimization has emerged at the forefront of expensive black-box optimization due to its data efficiency. Recent years have witnessed a proliferation of studies on the development of new Bayesian optimization algorithms and their applications. Hence, this paper attempts to provide a comprehensive and updated survey of recent advances in Bayesian optimization that are mainly based on Gaussian processes and identify challenging open problems. We categorize the existing work on Bayesian optimization into nine main groups according to the motivations and focus of the proposed algorithms. For each category, we present the main advances with respect to the construction of surrogate models and adaptation of the acquisition functions. Finally, we discuss the open questions and suggest promising future research directions, in particular with regard to heterogeneity, privacy preservation, and fairness in distributed and federated optimization systems.

What problem does this paper attempt to address?

The paper attempts to address key challenges in Bayesian Optimization (BO). Specifically, Bayesian Optimization is a sequential model optimization method that gradually optimizes the objective function by constructing a probabilistic model of the objective function (i.e., a surrogate model) and continuously updating this model after each new data sample. The paper mainly focuses on the following aspects: 1. **Model Construction and Update**: - How to effectively construct the initial probabilistic model. - How to efficiently update the model after acquiring new data. 2. **Exploration and Exploitation Balance**: - How to design and select appropriate Acquisition Functions (AFs) to balance between known high-fitness regions (exploitation) and under-explored regions (exploration). 3. **Multi-objective Optimization and Constraints**: - How to handle Multi-objective Optimization Problems (MOPs). - How to optimize in the presence of constraints (Constrained Bayesian Optimization, CBO). 4. **High-dimensional and Combinatorial Structure Optimization**: - How to effectively optimize in high-dimensional spaces. - How to handle Combinatorial Bayesian Optimization (COMBO). 5. **Computational Efficiency and Scalability**: - How to improve the computational efficiency of Bayesian Optimization, especially when facing expensive computational tasks. - How to scale Bayesian Optimization to large-scale problems and Multi-fidelity Optimization (MFO) and Multi-task Optimization (MTO). By addressing these issues, the paper aims to advance the performance and applicability of Bayesian Optimization in various practical applications, such as computational chemistry, materials science, and machine learning hyperparameter tuning.

Recent Advances in Bayesian Optimization

Pseudo-Bayesian Optimization

Bayesian optimization as a flexible and efficient design framework for sustainable process systems

Bayesian Optimization in Materials Science: A Survey

Simulation Based Bayesian Optimization

A Tutorial on Bayesian Optimization

Process Quality Control Through Bayesian Optimization with Adaptive Local Convergence

Bayesian Optimization using Deep Gaussian Processes

Transfer Learning for Bayesian Optimization: A Survey

Enhanced Bayesian Optimization via Preferential Modeling of Abstract Properties

A Bayesian optimization approach for data‐driven mixed‐integer nonlinear programming problems

Noisy Multiobjective Black-Box Optimization Using Bayesian Optimization.

Poisson Process for Bayesian Optimization

Bayesian Optimization with High-Dimensional Outputs

Practical Bayesian Optimization of Machine Learning Algorithms

A Data-Driven Bayesian Nonparametric Approach for Black-Box Optimization

Personalized Bayesian Optimization for Noisy Problems

Bayesian Optimisation for Constrained Problems

Mastering the exploration-exploitation trade-off in Bayesian Optimization

Bayesian Optimization Using Pseudo-Points

Deep Kernel Learning-Based Bayesian Optimization with Adaptive Kernel Functions