Abstract:Adaptive optimization methods such as AdaGrad, RMSprop and Adam have been proposed to achieve a rapid training process with an element-wise scaling term on learning rates. Though prevailing, they are observed to generalize poorly compared with SGD or even fail to converge due to unstable and extreme learning rates. Recent work has put forward some algorithms such as AMSGrad to tackle this issue but they failed to achieve considerable improvement over existing methods. In our paper, we demonstrate that extreme learning rates can lead to poor performance. We provide new variants of Adam and AMSGrad, called AdaBound and AMSBound respectively, which employ dynamic bounds on learning rates to achieve a gradual and smooth transition from adaptive methods to SGD and give a theoretical proof of convergence. We further conduct experiments on various popular tasks and models, which is often insufficient in previous work. Experimental results show that new variants can eliminate the generalization gap between adaptive methods and SGD and maintain higher learning speed early in training at the same time. Moreover, they can bring significant improvement over their prototypes, especially on complex deep networks. The implementation of the algorithm can be found at this https URL .

Back Propagation Algorithm Using Dynamic Optimal Learning Rate

Dynamic node creation and fast learning algorithm for a hybrid feedforward neural network

A Back-Propagation Algorithm with Adaptive Momentum Factor

Estimation Of Optimal Learning Parameters Based On Three-Term Backpropagation Algorithm

An Algorithm for Fast Convergence of Back Propagation by Enlarging Error

Global Convergence of Online Bp Training with Dynamic Learning Rate

Self-adaptive learning rates in backpropagation algorithm improve its function approximation performance

A direct adaptive method for faster backpropagation learning: the RPROP algorithm

Dynamical Optimal Learning For Fnn And Its Applications

A New Dynamic Optimal Learning Rate for a Two-Layer Neural Network

Adaptive Back-Propagation Algorithm with Magnified Error Signals

Optimal parameters selection of back propagation algorithm in the feedforward neural network

Back Propagation With Randomized Cost Function For Training Neural Networks

A synthetically backpropagation algorithm

An Improved Back-propagation NN Algorithm And Its Application

The Performance of the Backpropagation Algorithm with Varying Slope of the Activation Function

Training Feed-forward Neural Networks Using the Gradient Descent Method with the Optimal Stepsize

Adaptive Gradient Methods with Dynamic Bound of Learning Rate.

Meprop: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting

Optimal Algorithm for a Multilayer Feedforward Net and Its Application

Research on Back-Propagation Learning Algorithm Based on Weight Adjustment Parameters