Abstract:On High-Performance Computing (HPC) systems, several hyperparameter configurations can be evaluated in parallel to speed up the Hyperparameter Optimization (HPO) process. State-of-the-art HPO methods follow a bandit-based approach and build on top of successive halving, where the final performance of a combination is estimated based on a lower than fully trained fidelity performance metric and more promising combinations are assigned more resources over time. Frequently, the number of epochs is treated as a resource, letting more promising combinations train longer. Another option is to use the number of workers as a resource and directly allocate more workers to more promising configurations via data-parallel training. This article proposes a novel Resource-Adaptive Successive Doubling Algorithm (RASDA), which combines a resource-adaptive successive doubling scheme with the plain Asynchronous Successive Halving Algorithm (ASHA). Scalability of this approach is shown on up to 1,024 Graphics Processing Units (GPUs) on modern HPC systems. It is applied to different types of Neural Networks (NNs) and trained on large datasets from the Computer Vision (CV), Computational Fluid Dynamics (CFD), and Additive Manufacturing (AM) domains, where performing more than one full training run is usually infeasible. Empirical results show that RASDA outperforms ASHA by a factor of up to 1.9 with respect to the runtime. At the same time, the solution quality of final ASHA models is maintained or even surpassed by the implicit batch size scheduling of RASDA. With RASDA, systematic HPO is applied to a terabyte-scale scientific dataset for the first time in the literature, enabling efficient optimization of complex models on massive scientific data. The implementation of RASDA is available on <a class="link-external link-https" href="https://github.com/olympiquemarcel/rasda" rel="external noopener nofollow">this https URL</a>

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the challenges faced when performing hyperparameter optimization (HPO) of machine - learning models on large - scale datasets and high - performance computing systems (HPC). Specifically, it focuses on the following key issues: 1. **High - dimensional search space**: Finding the optimal combination of hyperparameters is a complex process because many different parameter combinations need to be evaluated. Each combination usually requires a complete training run to obtain its accuracy, which is very time - consuming. 2. **Efficient use of computing resources**: On high - performance computing systems, HPO can be accelerated by parallel processing, but how to efficiently allocate computing resources to reduce the total running time remains a challenge. 3. **Processing of large - scale datasets**: For large - scale datasets in fields such as computer vision (CV), computational fluid dynamics (CFD), and additive manufacturing (AM), performing multiple complete training runs is usually not feasible. Therefore, a method that can find the best hyperparameter configuration within a limited time is required. 4. **Limitations of existing methods**: Existing HPO methods such as the Asynchronous Successive Halving Algorithm (ASHA) mainly allocate resources by gradually halving in the time dimension, ignoring resource allocation in the spatial dimension (such as the number of GPUs). This leads to low efficiency in resource utilization, especially on large - scale datasets. To solve these problems, the paper proposes a new Resource - Adaptive Successive Doubling Algorithm (RASDA). RASDA combines gradual halving in the time dimension and gradual doubling in the spatial dimension, thereby using computing resources more effectively and finding the optimal hyperparameter configuration in a shorter time. ### Main features of RASDA - **Resource allocation combining time and space dimensions**: RASDA not only gradually eliminates poorly performing configurations according to performance (gradual halving), but also accelerates training by increasing the computing resources used by better - performing configurations (gradual doubling). - **Efficient use of large - scale datasets**: RASDA can perform hyperparameter optimization on datasets of up to 8.3TB, and is suitable for different types of neural networks such as convolutional neural networks (CNN), auto - encoders, and transformers. - **Comparison with existing methods**: Experimental results show that RASDA is 1.9 times faster than ASHA in running time while maintaining or exceeding the quality of the final model. ### Formula representation Some of the key formulas involved in the paper are as follows: - **Simple version of GNS (Gradient Noise Scale)**: \[ GNS_{\text{simple}}=\frac{\text{tr}(\Sigma)}{\vert G_{\text{true}}\vert^{2}} \] where $\Sigma$ is the gradient covariance matrix per data sample, and $G_{\text{true}}$ is the true gradient of full - batch gradient descent. - **Resource allocation in RASDA**: \[ \text{num\_rungs}=\left\lfloor\log_{s_f}\left(\frac{\max t}{\min t}\right)\right\rfloor \] \[ \text{rung\_milestones}=\min t\cdot s_f^{k} \] where $s_f$ is the scaling factor, and $k = 0,\ldots,\text{num\_rungs}$. Through these improvements, RASDA can efficiently perform hyperparameter optimization of large - scale datasets on high - performance computing systems, significantly shortening the running time of the optimization process.

Resource-Adaptive Successive Doubling for Hyperparameter Optimization with Large Datasets on High-Performance Computing Systems

Hyperparameter optimization of data-driven AI models on HPC systems

A System for Massively Parallel Hyperparameter Tuning

Multi-objective Asynchronous Successive Halving

A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUs

Adaptive Two-Stage Cloud Resource Scaling via Hierarchical Multi-Indicator Forecasting and Bayesian Decision-Making

Two-step hyperparameter optimization method: Accelerating hyperparameter search by using a fraction of a training dataset

Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF Surrogates

Efficient Hyperparameter Optimization for Deep Learning Algorithms Using Deterministic RBF Surrogates

Multi-Agent Deep Reinforcement Learning-Based Resource Allocation in HPC/AI Converged Cluster

Asynchronous Decentralized Bayesian Optimization for Large Scale Hyperparameter Optimization

Scaling Deep Learning on GPU and Knights Landing clusters

A Hybrid Parallelization Approach for Distributed and Scalable Deep Learning

Accelerating neural network training with distributed asynchronous and selective optimization (DASO)

Hierarchical Resource Partitioning on Modern GPUs: A Reinforcement Learning Approach

Hydra: A System for Large Multi-Model Deep Learning

Dynamic random mutation hybrid Harris hawk optimization and its application to training kernel extreme learning machine

ASA -- The Adaptive Scheduling Algorithm

Resource-Aware Pareto-Optimal Automated Machine Learning Platform

Accelerating Hyperparameter Optimization of Deep Neural Network via Progressive Multi-Fidelity Evaluation.

Benchmarking Resource Usage for Efficient Distributed Deep Learning