Abstract:The learning process and hyper-parameter optimization of artificial neural networks (ANNs) and deep learning (DL) architectures is considered one of the most challenging machine learning problems. Several past studies have used gradient-based back propagation methods to train DL architectures. However, gradient-based methods have major drawbacks such as stucking at local minimums in multi-objective cost functions, expensive execution time due to calculating gradient information with thousands of iterations and needing the cost functions to be continuous. Since training the ANNs and DLs is an NP-hard optimization problem, their structure and parameters optimization using the meta-heuristic (MH) algorithms has been considerably raised. MH algorithms can accurately formulate the optimal estimation of DL components (such as hyper-parameter, weights, number of layers, number of neurons, learning rate, etc.). This paper provides a comprehensive review of the optimization of ANNs and DLs using MH algorithms. In this paper, we have reviewed the latest developments in the use of MH algorithms in the DL and ANN methods, presented their disadvantages and advantages, and pointed out some research directions to fill the gaps between MHs and DL methods. Moreover, it has been explained that the evolutionary hybrid architecture still has limited applicability in the literature. Also, this paper classifies the latest MH algorithms in the literature to demonstrate their effectiveness in DL and ANN training for various applications. Most researchers tend to extend novel hybrid algorithms by combining MHs to optimize the hyper-parameters of DLs and ANNs. The development of hybrid MHs helps improving algorithms performance and capable of solving complex optimization problems. In general, the optimal performance of the MHs should be able to achieve a suitable trade-off between exploration and exploitation features. Hence, this paper tries to summarize various MH algorithms in terms of the convergence trend, exploration, exploitation, and the ability to avoid local minima. The integration of MH with DLs is expected to accelerate the training process in the coming few years. However, relevant publications in this way are still rare.

Towards optimal hierarchical training of neural networks

Adaptive Hierarchical Hyper-gradient Descent

Predict globally, correct locally: Parallel-in-time optimization of neural networks

Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptation

Further extensions on the successive approximation method for hierarchical optimal control problems and its application to learning

Training Deep Neural Networks by optimizing over nonlocal paths in hyperparameter space

Stochastic Hyperparameter Optimization through Hypernetworks

Navigating Scaling Laws: Compute Optimality in Adaptive Model Training

A successive approximation method in functional spaces for hierarchical optimal control problems and its application to learning

An effective algorithm for hyperparameter optimization of neural networks

Improving Generalization of Deep Neural Networks by Optimum Shifting

Training Artificial Neural Networks Using a Global Optimization Method That Utilizes Neural Networks

Adaptive, global, extended Kalman filters for training feedforward neural networks

Application of Meta-Heuristic Algorithms for Training Neural Networks and Deep Learning Architectures: A Comprehensive Review

Convergence Rates of Training Deep Neural Networks Via Alternating Minimization Methods.

Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions

On Convergence of Training Loss Without Reaching Stationary Points

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits

Adaptive Class Emergence Training: Enhancing Neural Network Stability and Generalization through Progressive Target Evolution

Adaptive Training Distributions with Scalable Online Bilevel Optimization