Abstract:Algorithm selection as well as hyperparameter optimization are tedious task that have to be dealt with when applying machine learning to real-world problems. Sequential model-based optimization (SMBO), based on so-called “surrogate models”, has been employed to allow for faster and more direct hyperparameter optimization. A surrogate model is a machine learning regression model which is trained on the meta-level instances in order to predict the performance of an algorithm on a specific data set given the hyperparameter settings and data set descriptors. Gaussian processes, for example, make good surrogate models as they provide probability distributions over labels. Recent work on SMBO also includes meta-data, i.e. observed hyperparameter performances on other data sets, into the process of hyperparameter optimization. This can, for example, be accomplished by learning transfer surrogate models on all available instances of meta-knowledge; however, the increasing amount of meta-information can make Gaussian processes infeasible, as they require the inversion of a large covariance matrix which grows with the number of instances. Consequently, instead of learning a joint surrogate model on all of the meta-data, we propose to learn individual surrogate models on the observations of each data set and then combine all surrogates to a joint one using ensembling techniques. The final surrogate is a weighted sum of all data set specific surrogates plus an additional surrogate that is solely learned on the target observations. Within our framework, any surrogate model can be used and explore Gaussian processes in this scenario. We present two different strategies for finding the weights used in the ensemble: the first is based on a probabilistic product of experts approach, and the second is based on kernel regression. Additionally, we extend the framework to directly estimate the acquisition function in the same setting, using a novel technique which we name the “transfer acquisition function”. In an empirical evaluation including comparisons to the current state-of-the-art on two publicly available meta-data sets, we are able to demonstrate that our proposed approach does not only scale to large meta-data, but also finds the stronger prediction models.

An Advanced and Robust Ensemble Surrogate Model: Extended Adaptive Hybrid Functions

Deep Residual Surrogate Model

MID-ESM: An Adaptive Median-Based Ensemble of Surrogate Models

Ensemble of surrogates in black-box-type engineering optimization: Recent advances and applications

A surrogate-ensemble assisted expensive many-objective optimization

Surrogate-Based Inverse Modeling of the Hydrological System: An Adaptive Approach Considering Surrogate Structural Error.

A surrogate-assisted hybrid swarm optimization algorithm for high-dimensional computationally expensive problems

A New Sequential Sampling Method for Surrogate Modeling Based on a Hybrid Metric

A Hybrid Variable-Fidelity Global Approximation Modelling Method Combining Tuned Radial Basis Function Base and Kriging Correction

Application of a Dirichlet Distribution-Based Ensemble Surrogate Model in Aerodynamic Optimization

A composite surrogate-assisted evolutionary algorithm for expensive many-objective optimization

A Hierarchical and Ensemble Surrogate-Assisted Evolutionary Algorithm With Model Reduction for Expensive Many-Objective Optimization

The Ensemble of Surrogate Model Based on Local and Global Errors

Surrogate-assisted Autoencoder-embedded Evolutionary Optimization Algorithm to Solve High-dimensional Expensive Problems

Efficient hierarchical surrogate-assisted differential evolution for high-dimensional expensive optimization

A novel multi-fidelity surrogate modeling framework integrated with sequential sampling criterion for non-hierarchical data

Multi-surrogate framework with an adaptive selection mechanism for production optimization

A two-stage infill strategy and surrogate-ensemble assisted expensive many-objective optimization

Scalable Gaussian process-based transfer surrogates for hyperparameter optimization

A multi-fidelity surrogate modeling method in the presence of non-hierarchical low-fidelity data

A Single-Fidelity Surrogate Modeling Method Based on Nonlinearity Integrated Multi-Fidelity Surrogate