Abstract:Hyperparameter optimization (HPO) is an important step in machine learning (ML) model development, but common practices are archaic -- primarily relying on manual or grid searches. This is partly because adopting advanced HPO algorithms introduces added complexity to the workflow, leading to longer computation times. This poses a notable challenge to ML applications, as suboptimal hyperparameter selections curtail the potential of ML model performance, ultimately obstructing the full exploitation of ML techniques. In this article, we present a two-step HPO method as a strategic solution to curbing computational demands and wait times, gleaned from practical experiences in applied ML parameterization work. The initial phase involves a preliminary evaluation of hyperparameters on a small subset of the training dataset, followed by a re-evaluation of the top-performing candidate models post-retraining with the entire training dataset. This two-step HPO method is universally applicable across HPO search algorithms, and we argue it has attractive efficiency gains. As a case study, we present our recent application of the two-step HPO method to the development of neural network emulators for aerosol activation. Although our primary use case is a data-rich limit with many millions of samples, we also find that using up to 0.0025% of the data (a few thousand samples) in the initial step is sufficient to find optimal hyperparameter configurations from much more extensive sampling, achieving up to 135-times speedup. The benefits of this method materialize through an assessment of hyperparameters and model performance, revealing the minimal model complexity required to achieve the best performance. The assortment of top-performing models harvested from the HPO process allows us to choose a high-performing model with a low inference cost for efficient use in global climate models (GCMs).

Meta Hyperparameter Optimization with Adversarial Proxy Subsets Sampling

Practical and sample efficient zero-shot HPO

Hyperparameter optimization: Classics, acceleration, online, multi-objective, and tools

HyperTuner: A Cross-Layer Multi-Objective Hyperparameter Auto-Tuning Framework for Data Analytic Services

Hyperparameter Optimization for Unsupervised Outlier Detection

Efficient hyperparameters optimization through model-based reinforcement learning with experience exploiting and meta-learning

Large Language Model Agent for Hyper-Parameter Optimization

HomOpt: A Homotopy-Based Hyperparameter Optimization Method

Two-step hyperparameter optimization method: Accelerating hyperparameter search by using a fraction of a training dataset

AntTune: An Efficient Distributed Hyperparameter Optimization System for Large-Scale Data.

Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm

HPS-Det: Dynamic Sample Assignment with Hyper-Parameter Search for Object Detection

TransBO: Hyperparameter Optimization Via Two-Phase Transfer Learning

Hyperparameter Optimization Can Even be Harmful in Off-Policy Learning and How to Deal with It

DP-HyPO: An Adaptive Private Hyperparameter Optimization Framework

In-the-loop Hyper-Parameter Optimization for LLM-Based Automated Design of Heuristics

Hyper-parameter Optimization for Federated Learning with Step-wise Adaptive Mechanism

Hierarchical Proxy Modeling for Improved HPO in Time Series Forecasting

MOFA: Modular Factorial Design for Hyperparameter Optimization

Learning an Explicit Hyperparameter Prediction Function Conditioned on Tasks

Efficient Meta Learning Via Minibatch Proximal Update