Abstract:Hyperparameter optimization (HPO) is an important step in machine learning (ML) model development, but common practices are archaic -- primarily relying on manual or grid searches. This is partly because adopting advanced HPO algorithms introduces added complexity to the workflow, leading to longer computation times. This poses a notable challenge to ML applications, as suboptimal hyperparameter selections curtail the potential of ML model performance, ultimately obstructing the full exploitation of ML techniques. In this article, we present a two-step HPO method as a strategic solution to curbing computational demands and wait times, gleaned from practical experiences in applied ML parameterization work. The initial phase involves a preliminary evaluation of hyperparameters on a small subset of the training dataset, followed by a re-evaluation of the top-performing candidate models post-retraining with the entire training dataset. This two-step HPO method is universally applicable across HPO search algorithms, and we argue it has attractive efficiency gains. As a case study, we present our recent application of the two-step HPO method to the development of neural network emulators for aerosol activation. Although our primary use case is a data-rich limit with many millions of samples, we also find that using up to 0.0025% of the data (a few thousand samples) in the initial step is sufficient to find optimal hyperparameter configurations from much more extensive sampling, achieving up to 135-times speedup. The benefits of this method materialize through an assessment of hyperparameters and model performance, revealing the minimal model complexity required to achieve the best performance. The assortment of top-performing models harvested from the HPO process allows us to choose a high-performing model with a low inference cost for efficient use in global climate models (GCMs).

A New Baseline for Automated Hyper-Parameter Optimization

Hyperparameters Adaptation for Restricted Boltzmann Machines Based on Free Energy

Simple Deterministic Selection-Based Genetic Algorithm for Hyperparameter Tuning of Machine Learning Models

A Modified Bayesian Optimization based Hyper-Parameter Tuning Approach for Extreme Gradient Boosting

Is One Hyperparameter Optimizer Enough?

Two-step hyperparameter optimization method: Accelerating hyperparameter search by using a fraction of a training dataset

Agent-based Collaborative Random Search for Hyper-parameter Tuning and Global Function Optimization

Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm

Adaptive Optimizer for Automated Hyperparameter Optimization Problem

Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020

Discrete Simulation Optimization for Tuning Machine Learning Method Hyperparameters

Hyperparameter Optimization: Comparing Genetic Algorithm against Grid Search and Bayesian Optimization

A Comparative Study of Hyperparameter Tuning Methods

Hyperparameter Tuning Algorithm Comparison with Machine Learning Algorithms

Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Features

Simple on-the-fly parameter selection mechanisms for two classical discrete black-box optimization benchmark problems

On the Effectiveness of Simple Success-Based Parameter Selection Mechanisms for Two Classical Discrete Black-Box Optimization Benchmark Problems

A Gradient-based Bilevel Optimization Approach for Tuning Hyperparameters in Machine Learning

Using sequential statistical tests for efficient hyperparameter tuning

Towards Stability of Parameter-free Optimization

Hyperparameter elegance: fine-tuning text analysis with enhanced genetic algorithm hyperparameter landscape