Abstract:Machine learning (ML) methods are used in most technical areas such as image recognition, product recommendation, financial analysis, medical diagnosis, and predictive maintenance. An important aspect of implementing ML methods involves controlling the learning process for the ML method so as to maximize the performance of the method under consideration. Hyperparameter tuning is the process of selecting a suitable set of ML method parameters that control its learning process. In this work, we demonstrate the use of discrete simulation optimization methods such as ranking and selection (R&S) and random search for identifying a hyperparameter set that maximizes the performance of a ML method. Specifically, we use the KN R&S method and the stochastic ruler random search method and one of its variations for this purpose. We also construct the theoretical basis for applying the KN method, which determines the optimal solution with a statistical guarantee via solution space enumeration. In comparison, the stochastic ruler method asymptotically converges to global optima and incurs smaller computational overheads. We demonstrate the application of these methods to a wide variety of machine learning models, including deep neural network models used for time series prediction and image classification. We benchmark our application of these methods with state-of-the-art hyperparameter optimization libraries such as $hyperopt$ and $mango$. The KN method consistently outperforms $hyperopt$'s random search (RS) and Tree of Parzen Estimators (TPE) methods. The stochastic ruler method outperforms the $hyperopt$ RS method and offers statistically comparable performance with respect to $hyperopt$'s TPE method and the $mango$ algorithm.

Hyperparameter Tuning of Deep learning Models in Keras

Genetic Algorithm Based Hyper-Parameter Tuning to Improve the Performance of Machine Learning Models

Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale

HyperTuner: A Cross-Layer Multi-Objective Hyperparameter Auto-Tuning Framework for Data Analytic Services

An empirical study of the impact of hyperparameter tuning and model optimization on the performance properties of deep neural networks

Discrete Simulation Optimization for Tuning Machine Learning Method Hyperparameters

Hyperparameter Tuning Algorithm Comparison with Machine Learning Algorithms

On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice

Deep Neural Network Hyperparameter Optimization with Orthogonal Array Tuning

Hyper-Parameter Optimization: A Review of Algorithms and Applications

HyperTendril: Visual Analytics for User-Driven Hyperparameter Optimization of Deep Neural Networks

Tune As You Scale: Hyperparameter Optimization For Compute Efficient Training

Is One Hyperparameter Optimizer Enough?

Optimizer Benchmarking Needs to Account for Hyperparameter Tuning

Auptimizer -- an Extensible, Open-Source Framework for Hyperparameter Tuning

Agent-based Collaborative Random Search for Hyper-parameter Tuning and Global Function Optimization

Impact of Parameter Tuning for Optimizing Deep Neural Network Models for Predicting Software Faults

Hyperparameter elegance: fine-tuning text analysis with enhanced genetic algorithm hyperparameter landscape

Hyperparameters in Reinforcement Learning and How To Tune Them