Abstract:Back propagation (BP) is widely used for parameter search of fully-connected layers in many neural networks. Although BP has the potential of quickly converging to a solution, due to its gradient-based nature, it tends to fall into a local optimum. Metaheuristics such as evolutionary computation (EC) techniques, as gradient-free methods, may have excellent global search capability due to their stochastic nature. However, these techniques tend to perform worse than BP in terms of convergence speed. In this paper, a hybrid gradient descent search algorithm (HGDSA) is proposed for training the parameters in fully-connected neural networks. HGDSA initially searches the space extensively by means of an ensemble of gradient descent strategies in the early stage and then uses BP as an exploitative local search operator. Moreover, a self-adaptive method which selects strategies and updates the learning rates of strategies has been designed and embedded in the global search operators to prevent stagnation in local optima. To verify the effectiveness of HGDSA, experiments were performed on eleven classification datasets. Experimental results demonstrate that the proposed HGDSA possesses both powerful global and local search abilities. Furthermore, the proposed approach appears to be promising also on high-dimensional datasets.

A Fast Hybrid Algorithm Of Global Optimization For Feedforward Neural Networks

Fast Learning Algorithms for Feedforward Neural Networks

Dynamic node creation and fast learning algorithm for a hybrid feedforward neural network

A Fast Learning Algorithm Of Neural Network For The Training And Recognition Of The Phonemes

A Novel Global Hybrid Algorithm for Feedforward Neural Networks

Optimal Algorithm for a Multilayer Feedforward Net and Its Application

A New Global Optimization Algorithm for Training Feedforward Neural Networks and Its Application

A Fast Learning Algorithm for Feedfoward Neural Network Based on the Layer-by-Layer and Neuron-by-Neuron Optimizing Procedure

A Novel Conjugate Gradient Method with Generalized Armijo Search for Efficient Training of Feedforward Neural Networks

A New Global Optimizing Algorithm for Fuzzy Neural Networks

A Global Optimization Algorithm Based on Filled-Function for Neural Networks

A Fast Learning Algorithm of Feedforward Neural Networks by Using Novel Error Functions

An Efficient Hybrid Incremental Algorithm for Complex-Valued Feedforward Neural Networks

A Fast Learning Strategy For Multilayer Feedforward Neural Networks

A L-BFGS Based Learning Algorithm for Complex-Valued Feedforward Neural Networks

A Global Optimization Algorithm For Training Feedforward Neural Networks

A Global Optimization Algorithm Based On Novel Interval Analysis For Training Neural Networks

New global learning algorithm for fuzzy neural networks based on modified quasi-newton method and genetic searching techniques

Adaptive Algorithm of Learning Rate for Feedforward Neural Network

A hybrid training algorithm based on gradient descent and evolutionary computation

A Novel Global Training Algorithm and Its Convergence Theorem for Fuzzy Neural Networks