Abstract:Hyperparameter optimization (HPO) is an important step in machine learning (ML) model development, but common practices are archaic -- primarily relying on manual or grid searches. This is partly because adopting advanced HPO algorithms introduces added complexity to the workflow, leading to longer computation times. This poses a notable challenge to ML applications, as suboptimal hyperparameter selections curtail the potential of ML model performance, ultimately obstructing the full exploitation of ML techniques. In this article, we present a two-step HPO method as a strategic solution to curbing computational demands and wait times, gleaned from practical experiences in applied ML parameterization work. The initial phase involves a preliminary evaluation of hyperparameters on a small subset of the training dataset, followed by a re-evaluation of the top-performing candidate models post-retraining with the entire training dataset. This two-step HPO method is universally applicable across HPO search algorithms, and we argue it has attractive efficiency gains. As a case study, we present our recent application of the two-step HPO method to the development of neural network emulators for aerosol activation. Although our primary use case is a data-rich limit with many millions of samples, we also find that using up to 0.0025% of the data (a few thousand samples) in the initial step is sufficient to find optimal hyperparameter configurations from much more extensive sampling, achieving up to 135-times speedup. The benefits of this method materialize through an assessment of hyperparameters and model performance, revealing the minimal model complexity required to achieve the best performance. The assortment of top-performing models harvested from the HPO process allows us to choose a high-performing model with a low inference cost for efficient use in global climate models (GCMs).

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: How to accelerate hyper - parameter optimization (HPO) in machine learning (ML) model development, so as to reduce the consumption of computing resources and waiting time, thereby improving the performance of machine - learning models. Specifically, traditional hyper - parameter optimization methods such as manual search or grid search are simple but inefficient, and modern advanced HPO algorithms are limited in their application due to the increased complexity of the work - flow and computing time. This not only hinders the full realization of the potential of machine - learning models but also restricts the wide application of machine - learning techniques in practical problems. For this reason, the author proposes a two - step hyper - parameter optimization method (two - step HPO method), aiming to significantly reduce the computing cost and accelerate the optimization process by using a small subset of the training data set for preliminary evaluation and then retraining the best - performing candidate models. The specific advantages of this method include: - **Computing efficiency**: By conducting a large number of trials on a small data subset first and then retraining the best - selected candidate models using the complete data set, the overall computing cost can be greatly reduced. - **Universal applicability**: This method is applicable to various hyper - parameter optimization algorithms and is independent of specific search algorithms or computing environments. - **Effect verification**: Through a case study of neural network simulators, the effectiveness of this method has been proven, especially when dealing with large - scale data sets, it can achieve a speed - up of up to 135 times. In summary, the core problem of this paper is to explore an efficient hyper - parameter optimization strategy to address the challenges of waste of computing resources and low optimization efficiency in existing methods.

Two-step hyperparameter optimization method: Accelerating hyperparameter search by using a fraction of a training dataset

Hyperparameter optimization: Classics, acceleration, online, multi-objective, and tools

Multi-level Training and Bayesian Optimization for Economical Hyperparameter Optimization

Hyper-Parameter Optimization: A Review of Algorithms and Applications

HomOpt: A Homotopy-Based Hyperparameter Optimization Method

Practical and sample efficient zero-shot HPO

Efficient Hyper-parameter Optimization for NLP Applications.

Hippo: Taming Hyper-parameter Optimization of Deep Learning with Stage Trees

HyperTuner: A Cross-Layer Multi-Objective Hyperparameter Auto-Tuning Framework for Data Analytic Services

AutoML with Parallel Genetic Algorithm for Fast Hyperparameters Optimization in Efficient IoT Time Series Prediction

Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm

Methodology for Hyperparameter Tuning of Deep Neural Networks for Efficient and Accurate Molecular Property Prediction

AntTune: An Efficient Distributed Hyperparameter Optimization System for Large-Scale Data.

Auptimizer -- an Extensible, Open-Source Framework for Hyperparameter Tuning

BTTackler: A Diagnosis-based Framework for Efficient Deep Learning Hyperparameter Optimization

Hyperopt: a Python library for model selection and hyperparameter optimization

Spend More to Save More (SM2): An Energy-Aware Implementation of Successive Halving for Sustainable Hyperparameter Optimization

PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning

Hyper-parameter Optimization for Federated Learning with Step-wise Adaptive Mechanism

Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep Learning