Abstract:Training neural networks by using conventional supervised backpropagation algorithms is a challenging task. This is due to significant limitations, such as the risk for local minimum stagnation in the loss landscape of neural networks. That may prevent the network from finding the global minimum of its loss function and therefore slow its convergence speed. Another challenge is the vanishing and exploding gradients that may happen when the gradients of the loss function of the model become either infinitesimally small or unmanageably large during the training. That also hinders the convergence of the neural models. On the other hand, the traditional gradient-based algorithms necessitate the pre-selection of learning parameters such as the learning rates, activation function, batch size, stopping criteria, and others. Recent research has shown the potential of evolutionary optimization algorithms to address most of those challenges in optimizing the overall performance of neural networks. In this research, we introduce and validate an evolutionary optimization framework to train multilayer perceptrons, which are simple feedforward neural networks. The suggested framework uses the recently proposed evolutionary cooperative optimization algorithm, namely, the dynamic group-based cooperative optimizer. The ability of this optimizer to solve a wide range of real optimization problems motivated our research group to benchmark its performance in training multilayer perceptron models. We validated the proposed optimization framework on a set of five datasets for engineering applications, and we compared its performance against the conventional backpropagation algorithm and other commonly used evolutionary optimization algorithms. The simulations showed the competitive performance of the proposed framework for most examined datasets in terms of overall performance and convergence. For three benchmarking datasets, the proposed framework provided increases of 2.7%, 4.83%, and 5.13% over the performance of the second best-performing optimizers, respectively.

A novel Mathematical Modeling for Deep Multilayer Perceptron Optimization: Architecture Optimization and Activation Functions Selection

Optimization of deep learning models: benchmark and analysis

Multilayer Perceptron optimization through Simulated Annealing and Fast Simulated Annealing

A Neural Network Model For General Minimax Problem

Optimum Scheme Selection for Multilayer Perceptron-Based Monte Carlo Simulation of Slope System Reliability

A Smooth Optimisation Perspective on Designing and Training Feedforward Multilayer Perceptrons

Manufacturing Process Modeling and Optimization Based on Multi-Layer Perceptron Network

Dynamic Optimization of Neural Network Structures Using Probabilistic Modeling

A Novel Input Variable Selection and Structure Optimization Algorithm for Multilayer Perceptron-Based Soft Sensors

Optimization inspired Multi-Branch Equilibrium Models

Ant Lion Optimizer: Theory, Literature Review, and Application in Multi-layer Perceptron Neural Networks

Multiobjective Optimization of Neural Network.

Training Multilayer Neural Network Based on Optimal Control Theory for Limited Computational Resources

A New Optimization Model for MLP Hyperparameter Tuning: Modeling and Resolution by Real-Coded Genetic Algorithm

Evolutionary optimization framework to train multilayer perceptrons for engineering applications

Genetic algorithm for neural network architecture optimization

Training Multi-Layer Perceptron with Enhanced Brain Storm Optimization Metaheuristics

Using Fitness Dependent Optimizer for Training Multi-layer Perceptron

An Efficient Optimization Technique for Training Deep Neural Networks

Building energy consumption prediction using multilayer perceptron neural network-assisted models; comparison of different optimization algorithms

Neural Networks Designing Neural Networks: Multi-Objective Hyper-Parameter Optimization