Abstract:Abstract Deep Neural Networks (DNNs) are widely regarded as the most effective learning tool for dealing with large datasets, and they have been successfully used in thousands of applications in a variety of fields. Based on these large datasets, they are trained to learn the relationships between various variables. The adaptive moment estimation (Adam) algorithm, a highly efficient adaptive optimization algorithm, is widely used as a learning algorithm in various fields for training DNN models. However, it needs to improve its generalization performance, especially when training with large-scale datasets. Therefore, in this paper, we propose HN Adam, a modified version of the Adam Algorithm, to improve its accuracy and convergence speed. The HN_Adam algorithm is modified by automatically adjusting the step size of the parameter updates over the training epochs. This automatic adjustment is based on the norm value of the parameter update formula according to the gradient values obtained during the training epochs. Furthermore, a hybrid mechanism was created by combining the standard Adam algorithm and the AMSGrad algorithm. As a result of these changes, the HN_Adam algorithm, like the stochastic gradient descent (SGD) algorithm, has good generalization performance and achieves fast convergence like other adaptive algorithms. To test the proposed HN_Adam algorithm performance, it is evaluated to train a deep convolutional neural network (CNN) model that classifies images using two different standard datasets: MNIST and CIFAR-10. The algorithm results are compared to the basic Adam algorithm and the SGD algorithm, in addition to other five recent SGD adaptive algorithms. In most comparisons, the HN Adam algorithm outperforms the compared algorithms in terms of accuracy and convergence speed. AdaBelief is the most competitive of the compared algorithms. In terms of testing accuracy and convergence speed (represented by the consumed training time), the HN-Adam algorithm outperforms the AdaBelief algorithm by an improvement of 1.0% and 0.29% for the MNIST dataset, and 0.93% and 1.68% for the CIFAR-10 dataset, respectively.

A Supervised Modified Hebbian Learning Method On Feed-forward Neural Networks

Dynamic node creation and fast learning algorithm for a hybrid feedforward neural network

Hebbian Deep Learning Without Feedback

Fast Learning Algorithms for Feedforward Neural Networks

Research of New Learning Method of Feedforward Neural Network

FastHebb: Scaling Hebbian Training of Deep Neural Networks to ImageNet Level

BackLink: Supervised Local Training with Backward Links

A Fast Hybrid Algorithm Of Global Optimization For Feedforward Neural Networks

An STDP-Based Supervised Learning Algorithm for Spiking Neural Networks

Improved Forward-Forward Contrastive Learning

Optimal Training of Feedforward Neural Networks Using Teaching-Learning-Based Optimization with Modified Learning Phases

Supervised perceptron learning vs unsupervised Hebbian unlearning: Approaching optimal memory retrieval in Hopfield-like networks

Supervised Learning in Multilayer Spiking Neural Networks With Spike Temporal Error Backpropagation

Training a Feed-forward Neural Network with Artificial Bee Colony Based Backpropagation Method

Scalable bio-inspired training of deep neural networks with FastHebb

Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry

A New Backpropagation Algorithm without Gradient Descent

A Modified Learning Algorithm for Improving the Fault Tolerance of BP Networks

Neuro-Modulated Hebbian Learning for Fully Test-Time Adaptation

A modified Adam algorithm for deep neural network optimization

A free from local minima algorithm for training regressive MLP neural networks