Abstract:Recent research has indicated a substantial rise in interest in understanding Nesterov's accelerated gradient methods via their continuous-time models. However, most existing studies focus on specific classes of Nesterov's methods, which hinders the attainment of an in-depth understanding and a unified perspective. To address this deficit, we present generalized continuous-time models that cover a broad range of Nesterov's methods, including those previously studied under existing continuous-time frameworks. Our key contributions are as follows. First, we identify the convergence rates of the generalized models, eliminating the need to determine the convergence rate for any specific continuous-time model derived from them. Second, we show that six existing continuous-time models are special cases of our generalized models, thereby positioning our framework as a unifying tool for analyzing and understanding these models. Third, we design a restart scheme for Nesterov's methods based on our generalized models and show that it ensures a monotonic decrease in objective function values. Owing to the broad applicability of our models, this scheme can be used to a broader class of Nesterov's methods compared to the original restart scheme. Fourth, we uncover a connection between our generalized models and gradient flow in continuous time, showing that the accelerated convergence rates of our generalized models can be attributed to a time reparametrization in gradient flow. Numerical experiment results are provided to support our theoretical analyses and results.

The Nesterov accelerated gradient algorithm for Auto-Regressive Exogenous models with random lost measurements: interpolation method and auxiliary model method

An Accelerating Method For Fuzzy C-Regression Models

A Method of ARX Model Estimation of Errors-in-Variables Systems

Adaptive Accelerated Proximal Gradient Algorithm for Auto-Regressive Exogenous Models with Outliers

Arx Model Estimation Of Multivariable Errors-In-Variables Systems

The application of natural neighbor interpolation in real-time environments

The expectation-maximization algorithm for autoregressive models with normal inverse Gaussian innovations

A Spatiotemporal Autoregressive Neural Network Interpolation Method for Discrete Environmental Factors

Gradient-based Parameter Estimation for a Nonlinear Exponential Autoregressive Time-series Model by Using the Multi-innovation

A NEW IDENTIFICATION APPROACH FOR ARX MODELS

A Generalized Spatial Autoregressive Neural Network Method for Three-Dimensional Spatial Interpolation

Generalized Continuous-Time Models for Nesterov's Accelerated Gradient Methods

Additional fractional gradient descent identification algorithm based on multi-innovation principle for autoregressive exogenous models

Adaptive Multi-Innovation Gradient Identification Algorithms for a Controlled Autoregressive Autoregressive Moving Average Model

Accelerated gradient methods for sparse statistical learning with nonconvex penalties

Data Filtering-Based Multi-innovation Stochastic Gradient Algorithm for Nonlinear Output Error Autoregressive Systems

Nesterov acceleration despite very noisy gradients

On a simultaneous parameter inference and missing data imputation for nonstationary autoregressive models

Novel Autoregressive Model Based On Adaptive Window-Extension And Patch-Geodesic Distance For Image Interpolation

Adaptive General Scale Interpolation Based on Weighted Autoregressive Models