Abstract:Artificial neural networks have achieved great success in artificial intelligence. Back propagation is one of the most widely used parameter search techniques for artificial neural networks, and it has been widely used in many deep learning approaches such as convolutional neural networks (CNNs), generator adversarial networks (GANs) and recurrent neural networks (RNNs) to train their fully-connected layers. However, the fully-connected layers of the neural networks with back propagation technique depends on the gradient descent strategy, which has the disadvantage of easily falling into local optima. Meanwhile, many evolutionary computation (EC) techniques are applied to the parameters optimization of neural networks. They take advantage of populations to achieve global search. But their convergence speed is relatively slow since they use their original evolutionary search strategies. In this paper, we propose a self-adaptive gradient descent search algorithm (SaGDSA) to search the parameters for fully-connected neural networks. Different from the existing adaptive gradient descent algorithm, the proposed algorithm does not require the user to have any prior knowledge and manually design the learning rate of different stages to match the different search stages. In addition, four kinds of gradient descent strategies are used to optimize parameters of fully-connected neural networks. All used datasets are collected from the University of California Irvine (UCI) machine learning repository. The experimental results indicate that SaGDSA is better than the comparison algorithms on most of the datasets. By introducing the self-adaptive mechanism, SaGDSA not only has excellent global search capabilities from evolutionary computation but also has local search capabilities from gradient descent.

A hybrid training algorithm based on gradient descent and evolutionary computation

Parallel adaptive hybrid genetic optimization algorithm and its application

Evolving Neural Networks Using the Hybrid of Ant Colony Optimization and Bp Algorithms

A self-adaptive gradient descent search algorithm for fully-connected neural networks

A Hybrid Learning Algorithm for the Optimization of Convolutional Neural Network.

Two Hybrid Learning Strategies of Feedforward Network

An Optimized BP Neuron Network Algorithm Based on GASA Hybrid Learning Strategy

Hybrid Cultural And Back-Propagation Algorithm For Neural Network Training

Hybrid Genetic Algorithms for Global Optimization Problems

An optimization method for neural network based on GA and TS algorithm

Hybridization of the multi-objective evolutionary algorithms and the gradient-based algorithms

Training Feed-forward Neural Networks Using the Gradient Descent Method with the Optimal Stepsize

An optimizing BP neural network algorithm based on genetic algorithm

A Hybrid Algorithm Based on Chips

A Hybrid GA-PSO Method for Evolving Architecture and Short Connections of Deep Convolutional Neural Networks

BGADAM: Boosting based Genetic-Evolutionary ADAM for Neural Network Optimization

Adaptively Transferring Deep Neural Networks with a Hybrid Evolution Strategy

Adaptive Stochastic Conjugate Gradient Optimization for Backpropagation Neural Networks

Hybrid Coordinate Descent for Efficient Neural Network Learning Using Line Search and Gradient Descent

A Hybrid Algorithm Based on Attention Model

A Hybrid Algorithm Based on Genetic Algorithm and Levenberg-Marquardt