Abstract:Abstract Deep Neural Networks (DNNs) are widely regarded as the most effective learning tool for dealing with large datasets, and they have been successfully used in thousands of applications in a variety of fields. Based on these large datasets, they are trained to learn the relationships between various variables. The adaptive moment estimation (Adam) algorithm, a highly efficient adaptive optimization algorithm, is widely used as a learning algorithm in various fields for training DNN models. However, it needs to improve its generalization performance, especially when training with large-scale datasets. Therefore, in this paper, we propose HN Adam, a modified version of the Adam Algorithm, to improve its accuracy and convergence speed. The HN_Adam algorithm is modified by automatically adjusting the step size of the parameter updates over the training epochs. This automatic adjustment is based on the norm value of the parameter update formula according to the gradient values obtained during the training epochs. Furthermore, a hybrid mechanism was created by combining the standard Adam algorithm and the AMSGrad algorithm. As a result of these changes, the HN_Adam algorithm, like the stochastic gradient descent (SGD) algorithm, has good generalization performance and achieves fast convergence like other adaptive algorithms. To test the proposed HN_Adam algorithm performance, it is evaluated to train a deep convolutional neural network (CNN) model that classifies images using two different standard datasets: MNIST and CIFAR-10. The algorithm results are compared to the basic Adam algorithm and the SGD algorithm, in addition to other five recent SGD adaptive algorithms. In most comparisons, the HN Adam algorithm outperforms the compared algorithms in terms of accuracy and convergence speed. AdaBelief is the most competitive of the compared algorithms. In terms of testing accuracy and convergence speed (represented by the consumed training time), the HN-Adam algorithm outperforms the AdaBelief algorithm by an improvement of 1.0% and 0.29% for the MNIST dataset, and 0.93% and 1.68% for the CIFAR-10 dataset, respectively.

Parameter Training Methods for Convolutional Neural Networks With Adaptive Adjustment Method Based on Borges Difference

Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentum

An Improved Reacceleration Optimization Algorithm Based on the Momentum Method for Image Recognition

Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization

Adaptive momentum with discriminative weight for neural network stochastic optimization

An Improved Adam Optimization Algorithm Combining Adaptive Coefficients and Composite Gradients Based on Randomized Block Coordinate Descent

Promoting Exploration in Memory-Augmented Adam using Critical Momenta

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

An automatic learning rate decay strategy for stochastic gradient descent optimization methods in neural networks

A novel adaptive momentum method for medical image classification using convolutional neural network

A new non-adaptive optimization method: Stochastic gradient descent with momentum and difference

Efficient Adaptive Optimization via Subset-Norm and Subspace-Momentum: Fast, Memory-Reduced Training with Convergence Guarantees

A modified Adam algorithm for deep neural network optimization

DEAM: Adaptive Momentum with Discriminative Weight for Stochastic Optimization

Conjugate-Gradient-like Based Adaptive Moment Estimation Optimization Algorithm for Deep Learning

A Convergent ADMM Framework for Efficient Neural Network Training

Adaptive Learning Rates with Maximum Variation Averaging.

A modification of adaptive moment estimation (adam) for machine learning

A New Method of PCNN's Parameter's Optimization

CoolMomentum: A Method for Stochastic Optimization by Langevin Dynamics with Simulated Annealing

An Improvement of Adam Based on a Cyclic Exponential Decay Learning Rate and Gradient Norm Constraints