Abstract:Deep learning model is a multi-layered network structure, and the network parameters that evaluate the final performance of the model must be trained by a deep learning optimizer. In comparison to the mainstream optimizers that utilize integer-order derivatives reflecting only local information, fractional-order derivatives optimizers, which can capture global information, are gradually gaining attention. However, relying solely on the long-term estimated gradients computed from fractional-order derivatives while disregarding the influence of recent gradients on the optimization process can sometimes lead to issues such as local optima and slower optimization speeds. In this paper, we design an adaptive learning rate optimizer called AdaGL based on the Grünwald–Letnikov (G–L) fractional-order derivative. It changes the direction and step size of parameter updating dynamically according to the long-term and short-term gradients information, addressing the problem of falling into local minima or saddle points. To be specific, by utilizing the global memory of fractional-order calculus, we replace the gradient of parameter update with G–L fractional-order approximated gradient, making better use of the long-term curvature information in the past. Furthermore, considering that the recent gradient information often impacts the optimization phase significantly, we propose a step size control coefficient to adjust the learning rate in real-time. To compare the performance of the proposed AdaGL with the current advanced optimizers, we conduct several different deep learning tasks, including image classification on CNNs, node classification and graph classification on GNNs, image generation on GANs, and language modeling on LSTM. Extensive experimental results demonstrate that AdaGL achieves stable and fast convergence, excellent accuracy, and good generalization performance.

Faster Secure Multiparty Computation of Adaptive Gradient Descent.

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

A(DP)$^2$2SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

A New Adaptive Gradient Method with Gradient Decomposition

Multiplexed gradient descent: Fast online training of modern datasets on hardware neural networks without backpropagation

AdaX: Adaptive Gradient Descent with Exponential Long Term Memory

Optimal Adaptive and Accelerated Stochastic Gradient Descent

SSGD: A Safe and Efficient Method of Gradient Descent

Adaptive Gradient Methods with Dynamic Bound of Learning Rate.

Low-Latency Privacy-Preserving Deep Learning Design via Secure MPC

GeoAdaLer: Geometric Insights into Adaptive Stochastic Gradient Descent Algorithms

An Adaptive Learning Rate Deep Learning Optimizer Using Long and Short-Term Gradients Based on G–L Fractional-Order Derivative

CADA: Communication-Adaptive Distributed Adam

Byzantine-Resilient Non-Convex Stochastic Gradient Descent

Multi-consensus Decentralized Accelerated Gradient Descent

Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective

Trend-Smooth: Accelerate Asynchronous SGD by Smoothing Parameters Using Parameter Trends

Reducing Adversarial Training Cost with Gradient Approximation

Momentum is All You Need for Data-Driven Adaptive Optimization

Global Convergence of Non-Convex Gradient Descent for Computing Matrix Squareroot

Asynchronous Accelerated Stochastic Gradient Descent.