Abstract:The learning process and hyper-parameter optimization of artificial neural networks (ANNs) and deep learning (DL) architectures is considered one of the most challenging machine learning problems. Several past studies have used gradient-based back propagation methods to train DL architectures. However, gradient-based methods have major drawbacks such as stucking at local minimums in multi-objective cost functions, expensive execution time due to calculating gradient information with thousands of iterations and needing the cost functions to be continuous. Since training the ANNs and DLs is an NP-hard optimization problem, their structure and parameters optimization using the meta-heuristic (MH) algorithms has been considerably raised. MH algorithms can accurately formulate the optimal estimation of DL components (such as hyper-parameter, weights, number of layers, number of neurons, learning rate, etc.). This paper provides a comprehensive review of the optimization of ANNs and DLs using MH algorithms. In this paper, we have reviewed the latest developments in the use of MH algorithms in the DL and ANN methods, presented their disadvantages and advantages, and pointed out some research directions to fill the gaps between MHs and DL methods. Moreover, it has been explained that the evolutionary hybrid architecture still has limited applicability in the literature. Also, this paper classifies the latest MH algorithms in the literature to demonstrate their effectiveness in DL and ANN training for various applications. Most researchers tend to extend novel hybrid algorithms by combining MHs to optimize the hyper-parameters of DLs and ANNs. The development of hybrid MHs helps improving algorithms performance and capable of solving complex optimization problems. In general, the optimal performance of the MHs should be able to achieve a suitable trade-off between exploration and exploitation features. Hence, this paper tries to summarize various MH algorithms in terms of the convergence trend, exploration, exploitation, and the ability to avoid local minima. The integration of MH with DLs is expected to accelerate the training process in the coming few years. However, relevant publications in this way are still rare.

Ant Lion Optimizer: Theory, Literature Review, and Application in Multi-layer Perceptron Neural Networks

An Optimized Ant System For Clustering With Elitist Ant And Local Search

Evolving Neural Networks Using the Hybrid of Ant Colony Optimization and Bp Algorithms

A Cognitively Inspired Hybridization of Artificial Bee Colony and Dragonfly Algorithms for Training Multi-layer Perceptrons

Ant Lion Optimizer Based on Spatial Information

Multi-objective ant lion optimizer: a multi-objective optimization algorithm for solving engineering problems

SHuffled Ant Lion Optimization approach with an exponentially weighted random walk strategy

Dolphin Swarm Extreme Learning Machine.

Competing leaders grey wolf optimizer and its application for training multi-layer perceptron classifier

Training Multi-Layer Perceptron with Enhanced Brain Storm Optimization Metaheuristics

Many-objective ant lion optimizer (MaOALO): A new many-objective optimizer with its engineering applications

Learning-Based Neural Ant Colony Optimization.

OWMA: An improved self-regulatory woodpecker mating algorithm using opposition-based learning and allocation of local memory for solving optimization problems

Using Fitness Dependent Optimizer for Training Multi-layer Perceptron

Dynamic Adaptive Ant Lion Optimizer applied to route planning for unmanned aerial vehicle

Application of Meta-Heuristic Algorithms for Training Neural Networks and Deep Learning Architectures: A Comprehensive Review

A novel hybrid antlion optimization algorithm for multi-objective task scheduling problems in cloud computing environments

Self-learning salp swarm algorithm for global optimization and its application in multi-layer perceptron model training

Evolutionary optimization framework to train multilayer perceptrons for engineering applications

Optimal Training of Feedforward Neural Networks Using Teaching-Learning-Based Optimization with Modified Learning Phases

A novel Mathematical Modeling for Deep Multilayer Perceptron Optimization: Architecture Optimization and Activation Functions Selection