Abstract:Choosing appropriate hyperparameters plays a crucial role in the success of neural networks as hyper-parameters directly control the behavior and performance of the training algorithms. To obtain efficient tuning, Bayesian optimization methods based on Gaussian process (GP) models are widely used. Despite numerous applications of Bayesian optimization in deep learning, the existing methodologies are developed based on a convenient but restrictive assumption that the tuning parameters are independent of each other. However, tuning parameters with conditional dependence are common in practice. In this paper, we focus on two types of them: branching and nested parameters. Nested parameters refer to those tuning parameters that exist only within a particular setting of another tuning parameter, and a parameter within which other parameters are nested is called a branching parameter. To capture the conditional dependence between branching and nested parameters, a unified Bayesian optimization framework is proposed. The sufficient conditions are rigorously derived to guarantee the validity of the kernel function, and the asymptotic convergence of the proposed optimization framework is proven under the continuum-armed-bandit setting. Based on the new GP model, which accounts for the dependent structure among input variables through a new kernel function, higher prediction accuracy and better optimization efficiency are observed in a series of synthetic simulations and real data applications of neural networks. Sensitivity analysis is also performed to provide insights into how changes in hyperparameter values affect prediction accuracy.

A Unified Gaussian Process for Branching and Nested Hyperparameter Optimization

A Method of Adaptive Hyperparameter Optimization for Deep Generative Models

Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization

An Efficient Bayesian Optimization Approach for Automated Optimization of Analog Circuits.

Provably Efficient Bayesian Optimization with Unbiased Gaussian Process Hyperparameter Estimation

Hyperparameters Optimization in Deep Convolutional Neural Network / Bayesian Approach with Gaussian Process Prior

Standard Gaussian Process Can Be Excellent for High-Dimensional Bayesian Optimization

Multi-level Training and Bayesian Optimization for Economical Hyperparameter Optimization

Practical Bayesian Optimization of Machine Learning Algorithms

Optimal Designs of Gaussian Processes with Budgets for Hyperparameter Optimization

Pre-training the Deep Generative Models with Adaptive Hyperparameter Optimization

Pseudo-Bayesian Optimization

A Study of Bayesian Neural Network Surrogates for Bayesian Optimization

Bayesian Hyperparameter Optimization with BoTorch, GPyTorch and Ax

Bayesian Optimization for Hyperparameters Tuning in Neural Networks

Distributed Gaussian Processes Hyperparameter Optimization for Big Data Using Proximal ADMM

Near-linear Time Gaussian Process Optimization with Adaptive Batching and Resparsification

Scalable Bayesian Optimization Using Vecchia Approximations of Gaussian Processes

Bayesian Optimization with High-Dimensional Outputs

Efficient Hyperparameter Optimization with Probability-based Resource Allocating on Deep Neural Networks

Reinforcement Learning via Gaussian Processes with Neural Network Dual Kernels