Abstract:Choosing appropriate hyperparameters plays a crucial role in the success of neural networks as hyper-parameters directly control the behavior and performance of the training algorithms. To obtain efficient tuning, Bayesian optimization methods based on Gaussian process (GP) models are widely used. Despite numerous applications of Bayesian optimization in deep learning, the existing methodologies are developed based on a convenient but restrictive assumption that the tuning parameters are independent of each other. However, tuning parameters with conditional dependence are common in practice. In this paper, we focus on two types of them: branching and nested parameters. Nested parameters refer to those tuning parameters that exist only within a particular setting of another tuning parameter, and a parameter within which other parameters are nested is called a branching parameter. To capture the conditional dependence between branching and nested parameters, a unified Bayesian optimization framework is proposed. The sufficient conditions are rigorously derived to guarantee the validity of the kernel function, and the asymptotic convergence of the proposed optimization framework is proven under the continuum-armed-bandit setting. Based on the new GP model, which accounts for the dependent structure among input variables through a new kernel function, higher prediction accuracy and better optimization efficiency are observed in a series of synthetic simulations and real data applications of neural networks. Sensitivity analysis is also performed to provide insights into how changes in hyperparameter values affect prediction accuracy.

Pre-training the Deep Generative Models with Adaptive Hyperparameter Optimization

A Method of Adaptive Hyperparameter Optimization for Deep Generative Models

Hyperparameters Adaptation for Restricted Boltzmann Machines Based on Free Energy

Bayesian Optimization Based on Pseudo Labels

Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization

Bayesian Hyperparameter Optimization with BoTorch, GPyTorch and Ax

Efficient Bayesian Optimization with Deep Kernel Learning and Transformer Pre-trained on Multiple Heterogeneous Datasets

A Unified Gaussian Process for Branching and Nested Hyperparameter Optimization

Multi-level Training and Bayesian Optimization for Economical Hyperparameter Optimization

Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep Learning

Adaptive Optimizer for Automated Hyperparameter Optimization Problem

Provably Efficient Bayesian Optimization with Unbiased Gaussian Process Hyperparameter Estimation

Optimal Designs of Gaussian Processes with Budgets for Hyperparameter Optimization

Hyperparameter optimization: Classics, acceleration, online, multi-objective, and tools

Efficient Hyper-parameter Optimization for NLP Applications.

In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization

Fast Model Selection and Hyperparameter Tuning for Generative Models

Combination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learning

Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm

Bayesian Optimization for Hyperparameters Tuning in Neural Networks

Efficient Hyperparameter Optimization for Deep Learning Algorithms Using Deterministic RBF Surrogates