Abstract:Parameter setting will have a great impact on overall behavior of a machine learning model in terms of training time, infrastructure resource requirements, model convergence, and model accuracy. While training machine learning models, it is very difficult to choose optimum values for various parameters to create the final model architecture. There are two types of parameters in machine learning model, one is referred as model parameters that are estimated by fitting the given data to the model. And the other is referred as model hyperparameters, these parameters are used to control the learning process. Model parameters are determined by machine ideally by exploration and automatically picks the optimum value; for example, the weights given to a neural network continuously update throughout each iteration until an optimal value is not reached. The method of hyperparameter tuning aims to determine the optimal combination of hyperparameters that will enable the model to function optimally. Setting the optimal mix of hyperparameters is the only method to maximize model performance. However, the designer is responsible for setting the hyperparameters that define the model architecture, such as the value of k in a kNN model, and the process of finding the optimum hyperparameter is referred to as hyperparameter tuning. Currently, this is handled in a variety of methods, including random searching of a specific solution space, sequential searching of the solution space using grids, and so on. In this article, comparative analysis of these methods to the genetic algorithm methodology for hyperparameter tuning is tested.

On Hyper-parameter Tuning for Stochastic Optimization Algorithms

Adaptive Structural Hyper-Parameter Configuration by Q-Learning

Variational Reinforcement Learning for Hyper-Parameter Tuning of Adaptive Evolutionary Algorithm

Random sampling-based automatic parameter tuning for nonlinear programming solvers

Agent-based Collaborative Random Search for Hyper-parameter Tuning and Global Function Optimization

Efficient hyperparameters optimization through model-based reinforcement learning with experience exploiting and meta-learning

Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale

Automated Machine Learning on Big Data using Stochastic Algorithm Tuning

Hyperparameters in Reinforcement Learning and How To Tune Them

Tuning-Free Stochastic Optimization

A Hybrid Approach To Parameter Tuning In Genetic Algorithms

Genetic Algorithm Based Hyper-Parameter Tuning to Improve the Performance of Machine Learning Models

Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization

A Survey of Automatic Parameter Tuning Methods for Metaheuristics

Towards Autonomous Reinforcement Learning: Automatic Setting of Hyper-parameters using Bayesian Optimization

Automated Configuration of Genetic Algorithms by Tuning for Anytime Performance

Stochastic Online Decisioning Hyper-Heuristic for High Dimensional Optimization

Fine-Tuning Adaptive Stochastic Optimizers: Determining the Optimal Hyperparameter $ε$ via Gradient Magnitude Histogram Analysis

No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RL

Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Features

Hyp-RL : Hyperparameter Optimization by Reinforcement Learning