Abstract:Hyperparameters are important for machine learning algorithms since they directly control the behaviors of training algorithms and have a significant effect on the performance of machine learning models. Several techniques have been developed and successfully applied for certain application domains. However, this work demands professional knowledge and expert experience. And sometimes it has to resort to the brute-force search. Therefore, if an efficient hyperparameter optimization algorithm can be developed to optimize any given machine learning method, it will greatly improve the efficiency of machine learning. In this paper, we consider building the relationship between the performance of the machine learning models and their hyperparameters by Gaussian processes. In this way, the hyperparameter tuning problem can be abstracted as an optimization problem and Bayesian optimization is used to solve the problem. Bayesian optimization is based on the Bayesian theorem. It sets a prior over the optimization function and gathers the information from the previous sample to update the posterior of the optimization function. A utility function selects the next sample point to maximize the optimization function. Several experiments were conducted on standard test datasets. Experiment results show that the proposed method can find the best hyperparameters for the widely used machine learning models, such as the random forest algorithm and the neural networks, even multi-grained cascade forest under the consideration of time cost.

BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters

Bayesian Hyperparameter Optimization with BoTorch, GPyTorch and Ax

High-dimensional Bayesian optimization using low-dimensional feature spaces

Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning

Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization

Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview

A Hybrid Monte Carlo Architecture for Parameter Optimization

Large-Batch, Iteration-Efficient Neural Bayesian Design Optimization

Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces

FlexHB: a More Efficient and Flexible Framework for Hyperparameter Optimization

Multi-level Training and Bayesian Optimization for Economical Hyperparameter Optimization

Combination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learning

PHOENICS: A universal deep Bayesian optimizer

Auptimizer -- an Extensible, Open-Source Framework for Hyperparameter Tuning

BayesOpt: A Library for Bayesian optimization with Robotics Applications

Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization

Multi-Fidelity Cost-Aware Bayesian Optimization

Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets

Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces

Bayesian Optimization with High-Dimensional Outputs

Bayesian Optimization For Multi-Objective Mixed-Variable Problems