Abstract:Machine learning has achieved remarkable success over the past couple of decades, often attributed to a combination of algorithmic innovations and the availability of high-quality data available at scale. However, a third critical component is the fine-tuning of hyperparameters, which plays a pivotal role in achieving optimal model performance. Despite its significance, hyperparameter optimization (HPO) remains a challenging task for several reasons. Many HPO techniques rely on naive search methods or assume that the loss function is smooth and continuous, which may not always be the case. Traditional methods, like grid search and Bayesian optimization, often struggle to quickly adapt and efficiently search the loss landscape. Grid search is computationally expensive, while Bayesian optimization can be slow to prime. Since the search space for HPO is frequently high-dimensional and non-convex, it is often challenging to efficiently find a global minimum. Moreover, optimal hyperparameters can be sensitive to the specific dataset or task, further complicating the search process. To address these issues, we propose a new hyperparameter optimization method, HomOpt, using a data-driven approach based on a generalized additive model (GAM) surrogate combined with homotopy optimization. This strategy augments established optimization methodologies to boost the performance and effectiveness of any given method with faster convergence to the optimum on continuous, discrete, and categorical domain spaces. We compare the effectiveness of HomOpt applied to multiple optimization techniques (e.g., Random Search, TPE, Bayes, and SMAC) showing improved objective performance on many standardized machine learning benchmarks and challenging open-set recognition tasks.

Adaptive Optimizer for Automated Hyperparameter Optimization Problem

Parallel adaptive hybrid genetic optimization algorithm and its application

A Method of Adaptive Hyperparameter Optimization for Deep Generative Models

Hyperparameter optimization: Classics, acceleration, online, multi-objective, and tools

Auptimizer -- an Extensible, Open-Source Framework for Hyperparameter Tuning

Hyperparameter Optimization: Comparing Genetic Algorithm against Grid Search and Bayesian Optimization

Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization

HomOpt: A Homotopy-Based Hyperparameter Optimization Method

Pre-training the Deep Generative Models with Adaptive Hyperparameter Optimization

Agent-based Collaborative Random Search for Hyper-parameter Tuning and Global Function Optimization

Adaptive Expansion Bayesian Optimization for Unbounded Global Optimization

Adaptive Optimization Algorithms for Machine Learning

Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm

Adaptive Local Bayesian Optimization Over Multiple Discrete Variables

Is One Hyperparameter Optimizer Enough?

Derivative-Free Optimization with Adaptive Experience for Efficient Hyper-Parameter Tuning.

HyperTuner: A Cross-Layer Multi-Objective Hyperparameter Auto-Tuning Framework for Data Analytic Services

A New Baseline for Automated Hyper-Parameter Optimization

Hyperopt: a Python library for model selection and hyperparameter optimization

Optimization-Driven Adaptive Experimentation

On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice