Abstract:Optimization algorithms with momentum have been widely used for building deep learning models because of the fast convergence rate. Momentum helps accelerate Stochastic gradient descent in relevant directions in parameter updating, minifying the oscillations of the parameters update route. The gradient of each step in optimization algorithms with momentum is calculated by a part of the training samples, so there exists stochasticity, which may bring errors to parameter updates. In this case, momentum placing the influence of the last step to the current step with a fixed weight is obviously inaccurate, which propagates the error and hinders the correction of the current step. Besides, such a hyperparameter can be extremely hard to tune in applications as well. In this paper, we introduce a novel optimization algorithm, namely, Discriminative wEight on Adaptive Momentum (DEAM). Instead of assigning the momentum term weight with a fixed hyperparameter, DEAM proposes to compute the momentum weight automatically based on the discriminative angle. The momentum term weight will be assigned with an appropriate value that configures momentum in the current step. In this way, DEAM involves fewer hyperparameters. DEAM also contains a novel backtrack term, which restricts redundant updates when the correction of the last step is needed. The backtrack term can effectively adapt the learning rate and achieve the anticipatory update as well. Extensive experiments demonstrate that DEAM can achieve a faster convergence rate than the existing optimization algorithms in training the deep learning models of both convex and nonconvex situations.

Fractional-order stochastic gradient descent method with momentum and energy for deep neural networks

A Novel Gradient Descent Optimizer based on Fractional Order Scheduler and its Application in Deep Neural Networks

Stochastic Momentum Method with Double Acceleration for Regularized Empirical Risk Minimization

Study on fast speed fractional order gradient descent method and its application in neural networks

Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent

Multi-stage stochastic gradient method with momentum acceleration

Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentum

Adaptive momentum with discriminative weight for neural network stochastic optimization

An Improved Reacceleration Optimization Algorithm Based on the Momentum Method for Image Recognition

A Unified Momentum-based Paradigm of Decentralized SGD for Non-Convex Models and Heterogeneous Data

Improving Deep Neural Networks' Training for Image Classification with Nonlinear Conjugate Gradient-Style Adaptive Momentum.

Inverse-Free Fast Natural Gradient Descent Method for Deep Learning

CoolMomentum: A Method for Stochastic Optimization by Langevin Dynamics with Simulated Annealing

The Improved Stochastic Fractional Order Gradient Descent Algorithm

Study on Two-Stage Fractional Order Gradient Descend Method

DEAM: Adaptive Momentum with Discriminative Weight for Stochastic Optimization

Signal Processing Meets SGD: From Momentum to Filter

Ordered Momentum for Asynchronous SGD

Design of Generalized Fractional Order Gradient Descent Method

Weighted Aggregating Stochastic Gradient Descent for Parallel Deep Learning

Convolutional Neural Networks with Fractional Order Gradient Method.