Abstract:Nowadays, different kinds of problems such as modeling, optimal control, and machine learning can be formulated as an optimization problem. Gradient descent is the most popular method to solve such problem and many accelerated gradient descents have been designed to improve the performance. In this paper, we will analyze the basic gradient descent, momentum gradient descent, and Nesterov accelerated gradient descent from the system perspective and it is found that all of them can be formulated as a feedback control problem for tracking an extreme point. On this basis, a unified gradient descent design procedure is given, where a high order transfer function is considered. Furthermore, as an extension, both a fractional integrator and a general fractional transfer function are considered, which resulting in the fractional gradient descent. Due to the infinite-dimensional property of fractional order systems, numerical inverse Laplace transform and Matlab command stmcb () are used to realize a finite-order implementation for the fractional gradient descent. Besides the simplified design procedure, it is found that the convergence rate of fractional gradient descent is more robust to the step size by simulating results.

Study on Two-Stage Fractional Order Gradient Descend Method

Study on fast speed fractional order gradient descent method and its application in neural networks

A Novel Fractional Order Speedest Gradient Descent Method and its application<sup>*</sup>

Generalization of the Gradient Method with Fractional Order Gradient Direction

Design of Generalized Fractional Order Gradient Descent Method

Study on Fractional Order Gradient Methods

Convolutional Neural Networks with Fractional Order Gradient Method.

The Improved Stochastic Fractional Order Gradient Descent Algorithm

A Sequential Descent Method for Global Optimization

Fractional Order Iteration for Gradient Descent Method Based on Event-Triggered Mechanism

An Effective and Efficient Two Stage Algorithm for Global Optimization

Fractional Order Gradient Methods for a General Class of Convex Functions

The Novel Adaptive Fractional Order Gradient Decent Algorithms Design via Robust Control

A novel perspective to gradient method: the fractional order approach

On The Unified Design Of Accelerated Gradient Descent

Applications of fractional calculus in learned optimization

A novel fractional order gradient method for identifying a linear system

Fractional-order stochastic gradient descent method with momentum and energy for deep neural networks

A Novel Gradient Descent Optimizer based on Fractional Order Scheduler and its Application in Deep Neural Networks

SSGD: A Safe and Efficient Method of Gradient Descent

Guaranteed Descent Conjugate Gradient Methods with Modified Secant Condition