Abstract:Artificial neural networks have achieved great success in artificial intelligence. Back propagation is one of the most widely used parameter search techniques for artificial neural networks, and it has been widely used in many deep learning approaches such as convolutional neural networks (CNNs), generator adversarial networks (GANs) and recurrent neural networks (RNNs) to train their fully-connected layers. However, the fully-connected layers of the neural networks with back propagation technique depends on the gradient descent strategy, which has the disadvantage of easily falling into local optima. Meanwhile, many evolutionary computation (EC) techniques are applied to the parameters optimization of neural networks. They take advantage of populations to achieve global search. But their convergence speed is relatively slow since they use their original evolutionary search strategies. In this paper, we propose a self-adaptive gradient descent search algorithm (SaGDSA) to search the parameters for fully-connected neural networks. Different from the existing adaptive gradient descent algorithm, the proposed algorithm does not require the user to have any prior knowledge and manually design the learning rate of different stages to match the different search stages. In addition, four kinds of gradient descent strategies are used to optimize parameters of fully-connected neural networks. All used datasets are collected from the University of California Irvine (UCI) machine learning repository. The experimental results indicate that SaGDSA is better than the comparison algorithms on most of the datasets. By introducing the self-adaptive mechanism, SaGDSA not only has excellent global search capabilities from evolutionary computation but also has local search capabilities from gradient descent.

A Recurrent RBF Neural Network Based on Modified Gravitational Search Algorithm

A Rbf Network Modeling Approach Combining Ga and Orthogonal Optimum Seeking Method

Estimation of catalyst activity using improved RBFN

Cuckoo Search Algorithm with Membrane Communication Mechanism for Modeling Overhead Crane Systems Using RBF Neural Networks.

DNA-NSGA-Ⅱ Nonlinear Dynamic System Modeling Approach Using RBF Neural Networks

A Rough RBF Neural Networks Optimized by the Genetic Algorithm

An optimizing method of RBF neural network based on genetic algorithm

RBF Neural Network Based on Q-Gaussian Function in Function Approximation.

MAR-GSA: Mixed attraction and repulsion based gravitational search algorithm

A Novel Algorithm for Global Optimization of RBF Neural Networks

Modelling of Electrohydraulic System Using RBF Neural Networks and Genetic Algorithm

The Growing Radial Basis Function (RBF) Neural Network and Its Applications.

An improved conjugate gradient algorithm for radial basis function (RBF) networks modelling

GA-Based RBFNN and its application in simultaneous determination of Fe, Mn, Cu and Zn

Improved Gravitational Search Algorithm Based on Adaptive Strategies.

An Adaptive Learning Algorithm Aimed at Improving RBF Network Generalization Ability

Efficient Training of RBF Networks Via the BYY Automated Model Selection Learning Algorithms

A self-adaptive gradient descent search algorithm for fully-connected neural networks

A Genetic Algorithm for Parameter Optimization of RBF Network

Parameter Optimization of RBF Networks Based on Evolutionary Model

RBF Network Optimization Based on K-Means and Quantum Genetic Algorithm