Abstract:Nowadays, different kinds of problems such as modeling, optimal control, and machine learning can be formulated as an optimization problem. Gradient descent is the most popular method to solve such problem and many accelerated gradient descents have been designed to improve the performance. In this paper, we will analyze the basic gradient descent, momentum gradient descent, and Nesterov accelerated gradient descent from the system perspective and it is found that all of them can be formulated as a feedback control problem for tracking an extreme point. On this basis, a unified gradient descent design procedure is given, where a high order transfer function is considered. Furthermore, as an extension, both a fractional integrator and a general fractional transfer function are considered, which resulting in the fractional gradient descent. Due to the infinite-dimensional property of fractional order systems, numerical inverse Laplace transform and Matlab command stmcb () are used to realize a finite-order implementation for the fractional gradient descent. Besides the simplified design procedure, it is found that the convergence rate of fractional gradient descent is more robust to the step size by simulating results.

Generalization of the Gradient Method with Fractional Order Gradient Direction

Design of Generalized Fractional Order Gradient Descent Method

Fractional Order Gradient Methods for a General Class of Convex Functions

Study on Fractional Order Gradient Methods

A novel perspective to gradient method: the fractional order approach

Convolutional Neural Networks with Fractional Order Gradient Method.

A Novel Fractional Order Speedest Gradient Descent Method and its application<sup>*</sup>

A novel fractional order gradient method for identifying a linear system

Convergence Analysis of Fractional Gradient Descent

Study on Two-Stage Fractional Order Gradient Descend Method

Study on fast speed fractional order gradient descent method and its application in neural networks

Applications of fractional calculus in learned optimization

On line parameter estimation based on gradient algorithm for fractional order systems.

Fractional Order Iteration for Gradient Descent Method Based on Event-Triggered Mechanism

An Integer-Fractional Gradient Algorithm for Back Propagation Neural Networks

The Novel Adaptive Fractional Order Gradient Decent Algorithms Design via Robust Control

Fractional gradient system and generalized Birkhoff system

The Improved Stochastic Fractional Order Gradient Descent Algorithm

On The Unified Design Of Accelerated Gradient Descent

Fractional gradient and its application to the fractional advection equation

A Novel Modified Fractional Order LMS Algorithm