Abstract:Deep learning is a sub-branch of artificial intelligence that acquires knowledge by training a neural network. It has many applications in the field of banking, automobile industry, agriculture, and healthcare industry. Deep learning has played a significant role in solving complex tasks related to computer vision, such as image classification, natural language processing, and object detection. On the other hand, optimizers also play an intrinsic role in training the deep learning model. Recent studies have proposed many deep learning models, such as VGG, ResNet, DenseNet, and ImageNet. In addition, there are many optimizers such as stochastic gradient descent (SGD), Adam, AdaDelta, Adabelief, and AdaMax. In this study, we have selected those models that require lower hardware requirements and shorter training times, which facilitates the overall training process. We have modified the Adam based optimizers and minimized the cyclic path. We have removed an additional hyper-parameter from RMSProp and observed that the optimizer works with various models. The learning rate is set to minimum and constant. The initial weights are updated after each epoch, which helps to improve the accuracy of the model. We also changed the position of the epsilon in the default Adam optimizer. By changing the position of the epsilon, it accumulates the updating process. We used various models with SGD, Adam, RMSProp, and the proposed optimization technique. The results indicate that the proposed method is effective in achieving the accuracy and works well with the state-of-the-art architectures.

Inertial Proximal Deep Learning Alternating Minimization for Efficient Neutral Network Training

Accelerated Gradient-free Neural Network Training by Multi-convex Alternating Optimization

A Learned Proximal Alternating Minimization Algorithm and Its Induced Network for a Class of Two-block Nonconvex and Nonsmooth Optimization

Convergence Rates of Training Deep Neural Networks Via Alternating Minimization Methods.

Effective Neural Network Training with a New Weighting Mechanism-Based Optimization Algorithm.

A Convergent ADMM Framework for Efficient Neural Network Training

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

AA-DLADMM: An Accelerated ADMM-based Framework for Training Deep Neural Networks

PathProx: A Proximal Gradient Algorithm for Weight Decay Regularized Deep Neural Networks

On Stochastic Primal-Dual Hybrid Gradient Approach for Compositely Regularized Minimization.

DANTE: Deep alternations for training neural networks

An Efficient Optimization Technique for Training Deep Neural Networks

Input Normalized Stochastic Gradient Descent Training of Deep Neural Networks

A Proximal Block Coordinate Descent Algorithm for Deep Neural Network Training

AGDA+: Proximal Alternating Gradient Descent Ascent Method With a Nonmonotone Adaptive Step-Size Search For Nonconvex Minimax Problems

A Natural Primal-Dual Hybrid Gradient Method for Adversarial Neural Network Training on Solving Partial Differential Equations

Win: Weight-Decay-Integrated Nesterov Acceleration for Faster Network Training.

Training Simplification and Model Simplification for Deep Learning : A Minimal Effort Back Propagation Method

PID Controller-Based Stochastic Optimization Acceleration for Deep Neural Networks

Reconstructing Deep Neural Networks: Unleashing the Optimization Potential of Natural Gradient Descent

Stochastic Gauss–Seidel type inertial proximal alternating linearized minimization and its application to proximal neural networks