Abstract:Recent research has indicated a substantial rise in interest in understanding Nesterov's accelerated gradient methods via their continuous-time models. However, most existing studies focus on specific classes of Nesterov's methods, which hinders the attainment of an in-depth understanding and a unified perspective. To address this deficit, we present generalized continuous-time models that cover a broad range of Nesterov's methods, including those previously studied under existing continuous-time frameworks. Our key contributions are as follows. First, we identify the convergence rates of the generalized models, eliminating the need to determine the convergence rate for any specific continuous-time model derived from them. Second, we show that six existing continuous-time models are special cases of our generalized models, thereby positioning our framework as a unifying tool for analyzing and understanding these models. Third, we design a restart scheme for Nesterov's methods based on our generalized models and show that it ensures a monotonic decrease in objective function values. Owing to the broad applicability of our models, this scheme can be used to a broader class of Nesterov's methods compared to the original restart scheme. Fourth, we uncover a connection between our generalized models and gradient flow in continuous time, showing that the accelerated convergence rates of our generalized models can be attributed to a time reparametrization in gradient flow. Numerical experiment results are provided to support our theoretical analyses and results.

Variable Fractional Order Nesterov Accelerated Gradient Algorithm for Rational Models with Missing Inputs Using Interpolated Model

The Nesterov accelerated gradient algorithm for Auto-Regressive Exogenous models with random lost measurements: interpolation method and auxiliary model method

Accelerated Identification Algorithms for Rational Models Based on the Vector Transformation

Biased Compensation Adaptive Gradient Algorithm for Rational Model with Time-Delay Using Self-Organising Maps

A Hybrid Variable-Fidelity Global Approximation Modelling Method Combining Tuned Radial Basis Function Base and Kriging Correction

A novel interpretation of Nesterov's acceleration via variable step-size linear multistep methods

A Novel Fractional Order Speedest Gradient Descent Method and its application<sup>*</sup>

The Gated Recurrent Unit Recurrent Neural Network Based Nesterov Accelerated Fractional Order Gradient Algorithm for SOC Estimation

An Accelerated Decentralized Stochastic Optimization Algorithm with Inexact Model

Additional fractional gradient descent identification algorithm based on multi-innovation principle for autoregressive exogenous models

A smoothing interval neural networks-based Caputo fractional-order gradient learning algorithm

Accelerating gradient descent and Adam via fractional gradients

On line parameter estimation based on gradient algorithm for fractional order systems.

Rational Empirical Interpolation Methods with Applications

Neural empirical interpolation method for nonlinear model reduction

The Novel Adaptive Fractional Order Gradient Decent Algorithms Design via Robust Control

A novel traffic flow prediction model: Variable order fractional grey model based on an improved grey evolution algorithm

Study on fast speed fractional order gradient descent method and its application in neural networks

A Novel Modified Fractional Order LMS Algorithm

Generalized Continuous-Time Models for Nesterov's Accelerated Gradient Methods

Fractional-Order Grey Prediction Method for Non-Equidistant Sequences