Abstract:Hyperparameter optimization aims to find the optimal hyperparameter configuration of a machine learning model, which provides the best performance on a validation dataset. Manual search usually leads to get stuck in a local hyperparameter configuration, and heavily depends on human intuition and experience. A simple alternative of manual search is random/grid search on a space of hyperparameters, which still undergoes extensive evaluations of validation errors in order to find its best configuration. Bayesian optimization that is a global optimization method for black-box functions is now popular for hyperparameter optimization, since it greatly reduces the number of validation error evaluations required, compared to random/grid search. Bayesian optimization generally finds the best hyperparameter configuration from random initialization without any prior knowledge. This motivates us to let Bayesian optimization start from the configurations that were successful on similar datasets, which are able to remarkably minimize the number of evaluations. In this paper, we propose deep metric learning to learn meta-features over datasets such that the similarity over them is effectively measured by Euclidean distance between their associated meta-features. To this end, we introduce a Siamese network composed of deep feature and meta-feature extractors, where deep feature extractor provides a semantic representation of each instance in a dataset and meta-feature extractor aggregates a set of deep features to encode a single representation over a dataset. Then, our learned meta-features are used to select a few datasets similar to the new dataset, so that hyperparameters in similar datasets are adopted as initializations to warm-start Bayesian hyperparameter optimization.

Combination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learning

Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization

A Method of Adaptive Hyperparameter Optimization for Deep Generative Models

Bayesian Optimization Based on Pseudo Labels

Hyperparameters Adaptation for Restricted Boltzmann Machines Based on Free Energy

PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning

Pre-training the Deep Generative Models with Adaptive Hyperparameter Optimization

Multi-level Training and Bayesian Optimization for Economical Hyperparameter Optimization

An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization

A Modified Bayesian Optimization based Hyper-Parameter Tuning Approach for Extreme Gradient Boosting

Hyperparameter Optimization: A Spectral Approach

Accelerating Hyperparameter Optimization of Deep Neural Network via Progressive Multi-Fidelity Evaluation.

Non-stationary Continuum-armed Bandits for Online Hyperparameter Optimization.

Cascaded Algorithm-Selection and Hyper-Parameter Optimization with Extreme-Region Upper Confidence Bound Bandit

Efficient Hyper-parameter Optimization for NLP Applications.

Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning

Learning to Warm-Start Bayesian Hyperparameter Optimization

Methodology for Hyperparameter Tuning of Deep Neural Networks for Efficient and Accurate Molecular Property Prediction

Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm

Optimal Designs of Gaussian Processes with Budgets for Hyperparameter Optimization