Abstract:The training or fine-tuning of machine learning, vision, and language models is often implemented as a pipeline: a sequence of stages encompassing data preparation, model training and evaluation. In this paper, we exploit pipeline structures to reduce the cost of hyperparameter tuning for model training/fine-tuning, which is particularly valuable for language models given their high costs in GPU-days. We propose a "memoization-aware" Bayesian Optimization (BO) algorithm, EEIPU, that works in tandem with a pipeline caching system, allowing it to evaluate significantly more hyperparameter candidates per GPU-day than other tuning algorithms. The result is better-quality hyperparameters in the same amount of search time, or equivalently, reduced search time to reach the same hyperparameter quality. In our benchmarks on machine learning (model ensembles), vision (convolutional architecture) and language (T5 architecture) pipelines, we compare EEIPU against recent BO algorithms: EEIPU produces an average of $103\%$ more hyperparameter candidates (within the same budget), and increases the validation metric by an average of $108\%$ more than other algorithms (where the increase is measured starting from the end of warm-up iterations).

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to reduce the cost of hyper - parameter tuning in the machine - learning, vision, and language - model training pipelines. Specifically, the paper focuses on how to reduce the cost of hyper - parameter tuning during model training or fine - tuning by leveraging the pipeline structure, especially for language models with high computational - resource requirements. The authors propose a "cache - aware" Bayesian optimization algorithm (EEIPU), which works in synergy with the pipeline cache system and can evaluate more hyper - parameter candidates per GPU - day, thereby achieving higher - quality hyper - parameters in the same amount of time or achieving the same hyper - parameter quality in less time. ### Main contributions: 1. **Proposing the EEIPU algorithm**: EEIPU is a cost - and cache - aware acquisition function. It extends the traditional Expected Improvement (EI) method, enabling it to adapt to multi - stage pipeline settings and handle cases with unknown costs. EEIPU reduces the search cost by combining the stage - cost model and the cache mechanism. 2. **Cost - cooling mechanism**: EEIPU introduces a cost - cooling factor, which gradually decreases as the remaining search budget decreases, encouraging the algorithm to explore low - cost areas in the early stage and shift to high - cost but unexplored areas in the later stage. 3. **Cache strategy**: The cache of EEIPU only retains the stage outputs of the top Q best - performing hyper - parameter configurations, which reduces the overhead time of EEIPU calculations and encourages the algorithm to explore high - performance, low - cost areas. ### Experimental results: - **Performance improvement**: In the benchmark tests, EEIPU evaluates on average 103% more hyper - parameter candidates than other algorithms under the same budget, and after warm - up iterations, it improves the validation metric by an average of 108%. - **Practical applications**: The paper conducts experiments on machine - learning (model integration), vision (convolutional architecture), and language (T5 architecture) pipelines, demonstrating the effectiveness of EEIPU. ### Core ideas of the solution: - **Cache mechanism**: By caching the outputs of intermediate stages, EEIPU can avoid re - running known stages, thereby saving computational resources. - **Cost - awareness**: EEIPU considers not only the expected improvement of hyper - parameter configurations but also their expected costs, enabling the algorithm to explore the hyper - parameter space more efficiently within a limited budget. ### Conclusion: This paper proposes an innovative method that significantly reduces the cost of hyper - parameter tuning by combining the cache mechanism and cost - aware Bayesian optimization, especially in the training of language models with high computational - resource requirements. This method performs well in multiple practical application scenarios and provides an effective solution for improving model - training efficiency.

Reducing Hyperparameter Tuning Costs in ML, Vision and Language Model Training Pipelines via Memoization-Awareness

Exploiting Reuse in Pipeline-Aware Hyperparameter Tuning

When Parameter-efficient Tuning Meets General-purpose Vision-language Models

MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism

PipeMoE: Accelerating Mixture-of-Experts through Adaptive Pipelining.

Efficient hyperparameters optimization through model-based reinforcement learning with experience exploiting and meta-learning

Parameter-efficient is not sufficient: Exploring Parameter, Memory, and Time Efficient Adapter Tuning for Dense Predictions

Tune As You Scale: Hyperparameter Optimization For Compute Efficient Training

Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applications

Superpipeline: A Universal Approach for Reducing GPU Memory Usage in Large Models

Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation

Cost-Efficient Online Hyperparameter Optimization

Two-step hyperparameter optimization method: Accelerating hyperparameter search by using a fraction of a training dataset

Parameter-efficient Tuning for Large Language Model Without Calculating Its Gradients

Asynchronous Batch Bayesian Optimization with Pipelining Evaluations for Experimental Resource$\unicode{x2013}$constrained Conditions

MPMoE: Memory Efficient MoE for Pre-Trained Models With Adaptive Pipeline Parallelism

PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Prediction

Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning

BIPEFT: Budget-Guided Iterative Search for Parameter Efficient Fine-Tuning of Large Pretrained Language Models

Full Parameter Fine-tuning for Large Language Models with Limited Resources

CoBABO: A Hyperparameter Search Method with Cost Budget Awareness