Abstract:Training Artificial Neural Networks poses a challenging and critical problem in machine learning. Despite the effectiveness of gradient-based learning methods, such as Stochastic Gradient Descent (SGD), in training neural networks, they do have several limitations. For instance, they require differentiable activation functions, and cannot optimize a model based on several independent non-differentiable loss functions simultaneously; for example, the F1-score, which is used during testing, can be used during training when a gradient-free optimization algorithm is utilized. Furthermore, the training in any DNN can be possible with a small size of the training dataset. To address these concerns, we propose an efficient version of the gradient-free Coordinate Search (CS) algorithm, an instance of General Pattern Search methods, for training neural networks. The proposed algorithm can be used with non-differentiable activation functions and tailored to multi-objective/multi-loss problems. Finding the optimal values for weights of ANNs is a large-scale optimization problem. Therefore instead of finding the optimal value for each variable, which is the common technique in classical CS, we accelerate optimization and convergence by bundling the weights. In fact, this strategy is a form of dimension reduction for optimization problems. Based on the experimental results, the proposed method, in some cases, outperforms the gradient-based approach, particularly, in situations with insufficient labeled training data. The performance plots demonstrate a high convergence rate, highlighting the capability of our suggested method to find a reasonable solution with fewer function calls. As of now, the only practical and efficient way of training ANNs with hundreds of thousands of weights is gradient-based algorithms such as SGD or Adam. In this paper we introduce an alternative method for training ANN.

A Training Algorithm with Selectable Search Direction for Complex-Valued Feedforward Neural Networks.

Cuckoo Search Algorithm with Membrane Communication Mechanism for Modeling Overhead Crane Systems Using RBF Neural Networks.

Dynamic node creation and fast learning algorithm for a hybrid feedforward neural network

A L-BFGS Based Learning Algorithm for Complex-Valued Feedforward Neural Networks

A Multilayer Complex Neural Network Training Algorithm and Its Application in Adaptive Equalization

Modelling of Electrohydraulic System Using RBF Neural Networks and Genetic Algorithm

A Complex Hebbian-type Learning Algorithm with Application to Complex Adaptive ⅡR Filtering

Heuristic directional search optimal path algorithm based on the variable raster model

Optimal Algorithm for a Multilayer Feedforward Net and Its Application

Anefficient Learning Algorithm For Binary Feedforward Neural Networks

A hybrid training algorithm based on gradient descent and evolutionary computation

Towards Optimally Efficient Tree Search with Deep Learning

Training Artificial Neural Networks by Coordinate Search Algorithm

A self-adaptive gradient descent search algorithm for fully-connected neural networks

A Novel Global Training Algorithm and Its Convergence Theorem for Fuzzy Neural Networks

A Fast Hybrid Algorithm Of Global Optimization For Feedforward Neural Networks

Partial Connection Based on Channel Attention for Differentiable Neural Architecture Search

A GA-based Algorithm for Searching Optimal Parameters in n-dimension Space

Hybrid Optimization Strategy for Radial Basis Function Neural Network Structure

Two Hybrid Learning Strategies of Feedforward Network

Differentiable Search for Finding Optimal Quantization Strategy