Abstract:Hyperparameter tuning is a critical function necessary for the effective deployment of most machine learning (ML) algorithms. It is used to find the optimal hyperparameter settings of an ML algorithm in order to improve its overall output performance. To this effect, several optimization strategies have been studied for fine-tuning the hyperparameters of many ML algorithms, especially in the absence of model-specific information. However, because most ML training procedures need a significant amount of computational time and memory, it is frequently necessary to build an optimization technique that converges within a small number of fitness evaluations. As a result, a simple deterministic selection genetic algorithm (SDSGA) is proposed in this article. The SDSGA was realized by ensuring that both chromosomes and their accompanying fitness values in the original genetic algorithm are selected in an elitist-like way. We assessed the SDSGA over a variety of mathematical test functions. It was then used to optimize the hyperparameters of two well-known machine learning models, namely, the convolutional neural network (CNN) and the random forest (RF) algorithm, with application on the MNIST and UCI classification datasets. The SDSGA’s efficiency was compared to that of the Bayesian Optimization (BO) and three other popular metaheuristic optimization algorithms (MOAs), namely, the genetic algorithm (GA), particle swarm optimization (PSO) and biogeography-based optimization (BBO) algorithms. The results obtained reveal that the SDSGA performed better than the other MOAs in solving 11 of the 17 known benchmark functions considered in our study. While optimizing the hyperparameters of the two ML models, it performed marginally better in terms of accuracy than the other methods while taking less time to compute.

Hyperparameter Optimization: Comparing Genetic Algorithm against Grid Search and Bayesian Optimization

Hyperparameter Tuning Algorithm Comparison with Machine Learning Algorithms

Parallel adaptive hybrid genetic optimization algorithm and its application

Genetic Algorithm Based Hyper-Parameter Tuning to Improve the Performance of Machine Learning Models

Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization

Hyperparameter Search for Machine Learning Algorithms for Optimizing the Computational Complexity

A Comparative Study of Hyperparameter Tuning Methods

A Modified Bayesian Optimization based Hyper-Parameter Tuning Approach for Extreme Gradient Boosting

On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice

Automated Configuration of Genetic Algorithms by Tuning for Anytime Performance

Simple Deterministic Selection-Based Genetic Algorithm for Hyperparameter Tuning of Machine Learning Models

Agent-based Collaborative Random Search for Hyper-parameter Tuning and Global Function Optimization

Practical Bayesian Optimization of Machine Learning Algorithms

Adaptive Optimizer for Automated Hyperparameter Optimization Problem

Is One Hyperparameter Optimizer Enough?

An effective algorithm for hyperparameter optimization of neural networks

SVM Parameter Optimization using Grid Search and Genetic Algorithm to Improve Classification Performance

A Gradient-based Bilevel Optimization Approach for Tuning Hyperparameters in Machine Learning

Hyperparameter elegance: fine-tuning text analysis with enhanced genetic algorithm hyperparameter landscape

Grid Search, Random Search, Genetic Algorithm: A Big Comparison for NAS