Abstract:Algorithm selection as well as hyperparameter optimization are tedious task that have to be dealt with when applying machine learning to real-world problems. Sequential model-based optimization (SMBO), based on so-called “surrogate models”, has been employed to allow for faster and more direct hyperparameter optimization. A surrogate model is a machine learning regression model which is trained on the meta-level instances in order to predict the performance of an algorithm on a specific data set given the hyperparameter settings and data set descriptors. Gaussian processes, for example, make good surrogate models as they provide probability distributions over labels. Recent work on SMBO also includes meta-data, i.e. observed hyperparameter performances on other data sets, into the process of hyperparameter optimization. This can, for example, be accomplished by learning transfer surrogate models on all available instances of meta-knowledge; however, the increasing amount of meta-information can make Gaussian processes infeasible, as they require the inversion of a large covariance matrix which grows with the number of instances. Consequently, instead of learning a joint surrogate model on all of the meta-data, we propose to learn individual surrogate models on the observations of each data set and then combine all surrogates to a joint one using ensembling techniques. The final surrogate is a weighted sum of all data set specific surrogates plus an additional surrogate that is solely learned on the target observations. Within our framework, any surrogate model can be used and explore Gaussian processes in this scenario. We present two different strategies for finding the weights used in the ensemble: the first is based on a probabilistic product of experts approach, and the second is based on kernel regression. Additionally, we extend the framework to directly estimate the acquisition function in the same setting, using a novel technique which we name the “transfer acquisition function”. In an empirical evaluation including comparisons to the current state-of-the-art on two publicly available meta-data sets, we are able to demonstrate that our proposed approach does not only scale to large meta-data, but also finds the stronger prediction models.

Surrogate Model Based Hyperparameter Tuning for Deep Learning with SPOT

Hyperparameter Tuning Cookbook: A guide for scikit-learn, PyTorch, river, and spotPython

In a Nutshell -- The Sequential Parameter Optimization Toolbox

SPOT: An R Package For Automatic and Interactive Tuning of Optimization Algorithms by Sequential Parameter Optimization

PyTorch Hyperparameter Tuning - A Tutorial for spotPython

Hyperparameter Tuning of Deep learning Models in Keras

SpotTune: Transfer Learning Through Adaptive Fine-Tuning

FastTuning: Enabling Fast and Efficient Hyper-Parameter Tuning with Partitioning and Parallelism of Search Space

Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale

Scalable Gaussian process-based transfer surrogates for hyperparameter optimization

Training Deep Neural Networks by optimizing over nonlocal paths in hyperparameter space

Discrete Simulation Optimization for Tuning Machine Learning Method Hyperparameters

Parallel Hyperparameter Optimization Of Spiking Neural Network

Simplifying Hyperparameter Tuning in Online Machine Learning -- The spotRiverGUI

Deep Parameterized Neural Networks for Hyperspectral Image Denoising

HyperTuner: A Cross-Layer Multi-Objective Hyperparameter Auto-Tuning Framework for Data Analytic Services

SpotTune: Leveraging Transient Resources for Cost-efficient Hyper-parameter Tuning in the Public Cloud

A Hybrid Sparrow Search Algorithm of the Hyperparameter Optimization in Deep Learning

Black-box and surrogate optimization for tuning spiking neural models of striatum plasticity

Two-step hyperparameter optimization method: Accelerating hyperparameter search by using a fraction of a training dataset

Fast Hyperparameter Optimization of Deep Neural Networks via Ensembling Multiple Surrogates.