Abstract:Hyper-parameter tuning (HPT) for deep learning (DL) models is prohibitively expensive. Sequential model-based optimization (SMBO) emerges as the state-of-the-art (SOTA) approach to automatically optimize HPT performance due to its heuristic advantages. Unfortunately, focusing on algorithm optimization rather than a large-scale parallel HPT system, existing SMBO-based approaches still cannot effectively remove their strong sequential nature, posing two performance problems: (1) extremely low tuning speed and (2) sub-optimal model quality . In this paper, we propose FastTuning, a fast, scalable, and generic system aiming at parallelly accelerating SMBO-based HPT for large DL/ML models. The key is to partition the highly complex search space into multiple smaller sub-spaces, each of which is assigned to and optimized by a different tuning worker in parallel. However, determining the right level of resource allocation to strike a balance between quality and cost remains a challenge. To address this, we further propose NIMBLE, a dynamic scheduling strategy that is specially designed for FastTuning, including (1) Dynamic Elimination Algorithm, (2) Sub-space Re-division, and (3) Posterior Information Sharing. Finally, we incorporate 6 SOTAs (i.e., 3 tuning algorithms and 3 parallel tuning tools) into FastTuning. Experimental results, on ResNet18, VGG19, ResNet50, and ResNet152, show that FastTuning can consistently offer much faster tuning speed (up to $80\times$ ) with better accuracy (up to 4.7% improvement), thereby enabling the application of automatic HPT to real-life DL models.

Deep Neural Network Hyperparameter Optimization with Orthogonal Array Tuning

Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale

Efficient Hyperparameter Optimization with Probability-based Resource Allocating on Deep Neural Networks

Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm

Hyper-Parameter Optimization: A Review of Algorithms and Applications

FastTuning: Enabling Fast and Efficient Hyper-Parameter Tuning with Partitioning and Parallelism of Search Space

An effective algorithm for hyperparameter optimization of neural networks

HyperNOMAD: Hyperparameter optimization of deep neural networks using mesh adaptive direct search

Training Deep Neural Networks by optimizing over nonlocal paths in hyperparameter space

Hyperparameter Tuning of Deep learning Models in Keras

Efficient Hyperparameter Optimization in Deep Learning Using a Variable Length Genetic Algorithm

Novel Suboptimal approaches for Hyperparameter Tuning of Deep Neural Network [under the shelf of Optical Communication]

Agent-based Collaborative Random Search for Hyper-parameter Tuning and Global Function Optimization

Fast Hyperparameter Optimization of Deep Neural Networks via Ensembling Multiple Surrogates.

Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization

Stage-based Hyper-parameter Optimization for Deep Learning

Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization

Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks

Hyperparameter optimization: Classics, acceleration, online, multi-objective, and tools

An empirical study of the impact of hyperparameter tuning and model optimization on the performance properties of deep neural networks

Bayesian Optimization for Hyperparameters Tuning in Neural Networks