Abstract:Choosing appropriate hyperparameters plays a crucial role in the success of neural networks as hyper-parameters directly control the behavior and performance of the training algorithms. To obtain efficient tuning, Bayesian optimization methods based on Gaussian process (GP) models are widely used. Despite numerous applications of Bayesian optimization in deep learning, the existing methodologies are developed based on a convenient but restrictive assumption that the tuning parameters are independent of each other. However, tuning parameters with conditional dependence are common in practice. In this paper, we focus on two types of them: branching and nested parameters. Nested parameters refer to those tuning parameters that exist only within a particular setting of another tuning parameter, and a parameter within which other parameters are nested is called a branching parameter. To capture the conditional dependence between branching and nested parameters, a unified Bayesian optimization framework is proposed. The sufficient conditions are rigorously derived to guarantee the validity of the kernel function, and the asymptotic convergence of the proposed optimization framework is proven under the continuum-armed-bandit setting. Based on the new GP model, which accounts for the dependent structure among input variables through a new kernel function, higher prediction accuracy and better optimization efficiency are observed in a series of synthetic simulations and real data applications of neural networks. Sensitivity analysis is also performed to provide insights into how changes in hyperparameter values affect prediction accuracy.

Efficient Hyperparameter Optimization with Probability-based Resource Allocating on Deep Neural Networks

Hyperparameters Adaptation for Restricted Boltzmann Machines Based on Free Energy

A Method of Adaptive Hyperparameter Optimization for Deep Generative Models

Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization

Efficient Hyperparameter Optimization for Deep Learning Algorithms Using Deterministic RBF Surrogates

Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF Surrogates

Pre-training the Deep Generative Models with Adaptive Hyperparameter Optimization

Generalized Population-Based Training for Hyperparameter Optimization in Reinforcement Learning

Deep Neural Network Hyperparameter Optimization with Orthogonal Array Tuning

Parameter Optimization with Conscious Allocation (POCA)

Practical Bayesian Optimization of Machine Learning Algorithms

Hyper-Parameter Optimization: A Review of Algorithms and Applications

Efficient Hyper-parameter Optimization for NLP Applications.

Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale

Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning

Hyperparameter optimization: Classics, acceleration, online, multi-objective, and tools

A Unified Gaussian Process for Branching and Nested Hyperparameter Optimization

A Modified Bayesian Optimization based Hyper-Parameter Tuning Approach for Extreme Gradient Boosting

Efficient hyperparameters optimization through model-based reinforcement learning with experience exploiting and meta-learning

Methodology for Hyperparameter Tuning of Deep Neural Networks for Efficient and Accurate Molecular Property Prediction

An effective algorithm for hyperparameter optimization of neural networks