Abstract:Hyperparameter optimization (HPO) is an important step in machine learning (ML) model development, but common practices are archaic -- primarily relying on manual or grid searches. This is partly because adopting advanced HPO algorithms introduces added complexity to the workflow, leading to longer computation times. This poses a notable challenge to ML applications, as suboptimal hyperparameter selections curtail the potential of ML model performance, ultimately obstructing the full exploitation of ML techniques. In this article, we present a two-step HPO method as a strategic solution to curbing computational demands and wait times, gleaned from practical experiences in applied ML parameterization work. The initial phase involves a preliminary evaluation of hyperparameters on a small subset of the training dataset, followed by a re-evaluation of the top-performing candidate models post-retraining with the entire training dataset. This two-step HPO method is universally applicable across HPO search algorithms, and we argue it has attractive efficiency gains. As a case study, we present our recent application of the two-step HPO method to the development of neural network emulators for aerosol activation. Although our primary use case is a data-rich limit with many millions of samples, we also find that using up to 0.0025% of the data (a few thousand samples) in the initial step is sufficient to find optimal hyperparameter configurations from much more extensive sampling, achieving up to 135-times speedup. The benefits of this method materialize through an assessment of hyperparameters and model performance, revealing the minimal model complexity required to achieve the best performance. The assortment of top-performing models harvested from the HPO process allows us to choose a high-performing model with a low inference cost for efficient use in global climate models (GCMs).

Evaluating Initialization of Nelder-Mead Method for Hyperparameter Optimization in Deep Learning

Effective hyperparameter optimization using Nelder-Mead method in deep learning

Learning to Warm-Start Bayesian Hyperparameter Optimization

Improvement of the Nelder-Mead method using Direct Inversion in Iterative Subspace

Influence of Initialization on the Performance of Metaheuristic Optimizers

Hybridizing Differential Evolution and Nelder-Mead Simplex Algorithm for Global Optimization

Exploring the Optimized Value of Each Hyperparameter in Various Gradient Descent Algorithms

Discrete Simulation Optimization for Tuning Machine Learning Method Hyperparameters

Two-step hyperparameter optimization method: Accelerating hyperparameter search by using a fraction of a training dataset

Effective Neural Network Training with a New Weighting Mechanism-Based Optimization Algorithm.

Improved Initialization Method for Metaheuristic Algorithms: A Novel Search Space View

How to Initialize your Network? Robust Initialization for WeightNorm & ResNets

Goal-Oriented Sensitivity Analysis of Hyperparameters in Deep Learning

Using Population Based Algorithms For Initializing Nonnegative Matrix Factorization

Training Deep Neural Networks by optimizing over nonlocal paths in hyperparameter space

Experienced Optimization with Reusable Directional Model for Hyper-Parameter Search

Improving Classification Performance in Dendritic Neuron Models through Practical Initialization Strategies

Multi-level Training and Bayesian Optimization for Economical Hyperparameter Optimization

Understanding the Role of Optimization in Double Descent

A multi-layer line search method to improve the initialization of optimization algorithms

Efficient Hyper-parameter Optimization for NLP Applications.