Abstract:Nowadays, different kinds of problems such as modeling, optimal control, and machine learning can be formulated as an optimization problem. Gradient descent is the most popular method to solve such problem and many accelerated gradient descents have been designed to improve the performance. In this paper, we will analyze the basic gradient descent, momentum gradient descent, and Nesterov accelerated gradient descent from the system perspective and it is found that all of them can be formulated as a feedback control problem for tracking an extreme point. On this basis, a unified gradient descent design procedure is given, where a high order transfer function is considered. Furthermore, as an extension, both a fractional integrator and a general fractional transfer function are considered, which resulting in the fractional gradient descent. Due to the infinite-dimensional property of fractional order systems, numerical inverse Laplace transform and Matlab command stmcb () are used to realize a finite-order implementation for the fractional gradient descent. Besides the simplified design procedure, it is found that the convergence rate of fractional gradient descent is more robust to the step size by simulating results.

Design of Generalized Fractional Order Gradient Descent Method

Generalization of the Gradient Method with Fractional Order Gradient Direction

Fractional Order Gradient Methods for a General Class of Convex Functions

A Novel Fractional Order Speedest Gradient Descent Method and its application<sup>*</sup>

Study on Fractional Order Gradient Methods

On The Unified Design Of Accelerated Gradient Descent

A novel perspective to gradient method: the fractional order approach

A novel fractional order gradient method for identifying a linear system

Convergence Analysis of Fractional Gradient Descent

The Novel Adaptive Fractional Order Gradient Decent Algorithms Design via Robust Control

Fractional Order Iteration for Gradient Descent Method Based on Event-Triggered Mechanism

The Improved Stochastic Fractional Order Gradient Descent Algorithm

Convolutional Neural Networks with Fractional Order Gradient Method.

Generalized Finite Difference Method for a Class of Multidimensional Space-Fractional Diffusion Equations

Study on Two-Stage Fractional Order Gradient Descend Method

Study on fast speed fractional order gradient descent method and its application in neural networks

Numerical Solutions of Fractional Advection–diffusion Equations with a Kind of New Generalized Fractional Derivative

Discussion on Extreme Points with Fractional Order Derivatives

On line parameter estimation based on gradient algorithm for fractional order systems.

Applications of fractional calculus in learned optimization

The Design and Performance Analysis of Multivariate Fractional-Order Gradient-Based Extremum Seeking Approach