Abstract:Deep learning model is a multi-layered network structure, and the network parameters that evaluate the final performance of the model must be trained by a deep learning optimizer. In comparison to the mainstream optimizers that utilize integer-order derivatives reflecting only local information, fractional-order derivatives optimizers, which can capture global information, are gradually gaining attention. However, relying solely on the long-term estimated gradients computed from fractional-order derivatives while disregarding the influence of recent gradients on the optimization process can sometimes lead to issues such as local optima and slower optimization speeds. In this paper, we design an adaptive learning rate optimizer called AdaGL based on the Grünwald–Letnikov (G–L) fractional-order derivative. It changes the direction and step size of parameter updating dynamically according to the long-term and short-term gradients information, addressing the problem of falling into local minima or saddle points. To be specific, by utilizing the global memory of fractional-order calculus, we replace the gradient of parameter update with G–L fractional-order approximated gradient, making better use of the long-term curvature information in the past. Furthermore, considering that the recent gradient information often impacts the optimization phase significantly, we propose a step size control coefficient to adjust the learning rate in real-time. To compare the performance of the proposed AdaGL with the current advanced optimizers, we conduct several different deep learning tasks, including image classification on CNNs, node classification and graph classification on GNNs, image generation on GANs, and language modeling on LSTM. Extensive experimental results demonstrate that AdaGL achieves stable and fast convergence, excellent accuracy, and good generalization performance.

An Integer-Fractional Gradient Algorithm for Back Propagation Neural Networks

Convolutional Neural Networks with Fractional Order Gradient Method.

Convergence Analysis of Novel Fractional-Order Backpropagation Neural Networks With Regularization Terms

Propagating Asymptotic-Estimated Gradients for Low Bitwidth Quantized Neural Networks

Generalization of the Gradient Method with Fractional Order Gradient Direction

Study on fast speed fractional order gradient descent method and its application in neural networks

A smoothing interval neural networks-based Caputo fractional-order gradient learning algorithm

An Adaptive Learning Rate Deep Learning Optimizer Using Long and Short-Term Gradients Based on G–L Fractional-Order Derivative

Applications of fractional calculus in learned optimization

Design of Generalized Fractional Order Gradient Descent Method

Study on Fractional Order Gradient Methods

A Novel Gradient Descent Optimizer based on Fractional Order Scheduler and its Application in Deep Neural Networks

A Novel Fractional Order Speedest Gradient Descent Method and its application<sup>*</sup>

The Improved Stochastic Fractional Order Gradient Descent Algorithm

Fast Learning Algorithms for Feedforward Neural Networks

Fractional-order spike-timing-dependent gradient descent for multi-layer spiking neural networks

Fractional Hopfield Neural Networks: Fractional Dynamic Associative Recurrent Neural Networks

Deep neural network methods for solving forward and inverse problems of time fractional diffusion equations with conformable derivative

Application of Nonlinear Fractional Differential Equations in Computer Artificial Intelligence Algorithms

Exploring the Effects of Caputo Fractional Derivative in Spiking Neural Network Training

Fractional-order stochastic gradient descent method with momentum and energy for deep neural networks