Abstract:Stochastic Gradient Decent (SGD) is one of the core techniques behind the success of deep neural networks. The gradient provides information on the direction in which a function has the steepest rate of change. The main problem with basic SGD is to change by equal sized steps for all parameters, irrespective of gradient behavior. Hence, an efficient way of deep network optimization is to make adaptive step sizes for each parameter. Recently, several attempts have been made to improve gradient descent methods such as AdaGrad, AdaDelta, RMSProp and Adam. These methods rely on the square roots of exponential moving averages of squared past gradients. Thus, these methods do not take advantage of local change in gradients. In this paper, a novel optimizer is proposed based on the difference between the present and the immediate past gradient (i.e., diffGrad). In the proposed diffGrad optimization technique, the step size is adjusted for each parameter in such a way that it should have a larger step size for faster gradient changing parameters and a lower step size for lower gradient changing parameters. The convergence analysis is done using the regret bound approach of online learning framework. Rigorous analysis is made in this paper over three synthetic complex non-convex functions. The image categorization experiments are also conducted over the CIFAR10 and CIFAR100 datasets to observe the performance of diffGrad with respect to the state-of-the-art optimizers such as SGDM, AdaGrad, AdaDelta, RMSProp, AMSGrad, and Adam. The residual unit (ResNet) based Convolutional Neural Networks (CNN) architecture is used in the experiments. The experiments show that diffGrad outperforms other optimizers. Also, we show that diffGrad performs uniformly well for training CNN using different activation functions. The source code is made publicly available at <a class="link-external link-https" href="https://github.com/shivram1987/diffGrad" rel="external noopener nofollow">this https URL</a>.

A Hessian-Free Gradient Flow (HFGF) method for the optimisation of deep learning neural networks

Efficient and stable SAV-based methods for gradient flows arising from deep learning

Neural Sinkhorn Gradient Flow

Inverse-Free Fast Natural Gradient Descent Method for Deep Learning

diffGrad: An Optimization Method for Convolutional Neural Networks

Reconstructing Deep Neural Networks: Unleashing the Optimization Potential of Natural Gradient Descent

Block-diagonal Hessian-free Optimization for Training Neural Networks

Fixed-Time Gradient Flows for Solving Constrained Optimization: A Unified Approach

A Distributed Optimisation Framework Combining Natural Gradient with Hessian-Free for Discriminative Sequence Training

Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training

Gradient flows and proximal splitting methods: A unified view on accelerated and stochastic optimization

Hybrid Quantum-Classical Scheduling for Accelerating Neural Network Training with Newton's Gradient Descent

Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobians

Gradient Descent: The Ultimate Optimizer

A gradient descent akin method for constrained optimization: algorithms and applications

Accelerating Hessian-free optimization for deep neural networks by implicit preconditioning and sampling

Fast Unconstrained Optimization via Hessian Averaging and Adaptive Gradient Sampling Methods

Parameterized Wasserstein Gradient Flow

Projected Forward Gradient-Guided Frank-Wolfe Algorithm via Variance Reduction

Normalized gradient flow optimization in the training of ReLU artificial neural networks

Large-Scale Wasserstein Gradient Flows