Abstract:The growing ubiquity of machine learning (ML) has led it to enter various areas of computer science, including black-box optimization (BBO). Recent research is particularly concerned with Bayesian optimization (BO). BO-based algorithms are popular in the ML community, as they are used for hyperparameter optimization and more generally for algorithm configuration. However, their efficiency decreases as the dimensionality of the problem and the budget of evaluations increase. Meanwhile, derivative-free optimization methods have evolved independently in the optimization community. Therefore, we urge to understand whether cross-fertilization is possible between the two communities, ML and BBO, i.e., whether algorithms that are heavily used in ML also work well in BBO and vice versa. Comparative experiments often involve rather small benchmarks and show visible problems in the experimental setup, such as poor initialization of baselines, overfitting due to problem-specific setting of hyperparameters, and low statistical significance. With this paper, we update and extend a comparative study presented by Hutter et al. in 2013. We compare BBO tools for ML with more classical heuristics, first on the well-known BBOB benchmark suite from the COCO environment and then on Direct Policy Search for OpenAI Gym, a reinforcement learning benchmark. Our results confirm that BO-based optimizers perform well on both benchmarks when budgets are limited, albeit with a higher computational cost, while they are often outperformed by algorithms from other families when the evaluation budget becomes larger. We also show that some algorithms from the BBO community perform surprisingly well on ML tasks.

An evaluation of sequential model-based optimization for expensive blackbox functions

Sequential model based optimization of partially defined functions under unknown constraints

mlrMBO: A Modular Framework for Model-Based Optimization of Expensive Black-Box Functions

Sequential vs. Integrated Algorithm Selection and Configuration: A Case Study for the Modular CMA-ES

MA-BBOB: Many-Affine Combinations of BBOB Functions for Evaluating AutoML Approaches in Noiseless Numerical Black-Box Optimization Contexts

Comparison of High-Dimensional Bayesian Optimization Algorithms on BBOB

Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gym

An Empirical Review of Model-based Adaptive Sampling for Global Optimization of Expensive Black-box Functions

MA-BBOB: A Problem Generator for Black-Box Optimization Using Affine Combinations and Shifts

B2Opt: Learning to Optimize Black-box Optimization with Little Budget

Impact of Training Instance Selection on Automated Algorithm Selection Models for Numerical Black-box Optimization

An Improved Surrogate Based Optimization Method for Expensive Black-box Problems

Blackbox: A procedure for parallel optimization of expensive black-box functions

PMBO: Enhancing Black-Box Optimization through Multivariate Polynomial Surrogates

A Comparative Study of Large-Scale Variants of CMA-ES

Framework and Benchmarks for Combinatorial and Mixed-variable Bayesian Optimization

Practical Batch Bayesian Optimization for Less Expensive Functions

Versatile Black-Box Optimization

PI is back! Switching Acquisition Functions in Bayesian Optimization

Multi-Step Budgeted Bayesian Optimization with Unknown Evaluation Costs

OpenBox: A Generalized Black-box Optimization Service