Abstract:This paper investigates asymptotic behaviors of gradient descent algorithms (particularly accelerated gradient descent and stochastic gradient descent) in the context of stochastic optimization arising in statistics and machine learning where objective functions are estimated from available data. We show that these algorithms can be computationally modeled by continuous-time ordinary or stochastic differential equations. We establish gradient flow central limit theorems to describe the limiting dynamic behaviors of these computational algorithms and the large-sample performances of the related statistical procedures, as the number of algorithm iterations and data size both go to infinity, where the gradient flow central limit theorems are governed by some linear ordinary or stochastic differential equations like time-dependent Ornstein-Uhlenbeck processes. We illustrate that our study can provide a novel unified framework for a joint computational and statistical asymptotic analysis, where the computational asymptotic analysis studies dynamic behaviors of these algorithms with the time (or the number of iterations in the algorithms), the statistical asymptotic analysis investigates large sample behaviors of the statistical procedures (like estimators and classifiers) that the algorithms are applied to compute, and in fact the statistical procedures are equal to the limits of the random sequences generated from these iterative algorithms as the number of iterations goes to infinity. The joint analysis results based on the obtained gradient flow central limit theorems can identify four factors - learning rate, batch size, gradient covariance, and Hessian - to derive new theory regarding the local minima found by stochastic gradient descent for solving non-convex optimization problems.

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization

Asynchronous Proximal Stochastic Gradient Algorithm for Composition Optimization Problems

Asynchronous Accelerated Stochastic Gradient Descent.

Fast Asynchronous Parallel Stochastic Gradient Descent: A Lock-Free Approach With Convergence Guarantee

Parallel Asynchronous Stochastic Variance Reduction for Nonconvex Optimization

Fast Asynchronous Parallel Stochastic Gradient Decent

AsGrad: A Sharp Unified Analysis of Asynchronous-SGD Algorithms

Asynchronous Stochastic Proximal Methods for Nonconvex Nonsmooth Optimization.

Differential Equations for Modeling Asynchronous Algorithms

Accelerating Asynchronous Algorithms for Convex Optimization by Momentum Compensation

A Sharp Convergence Rate for the Asynchronous Stochastic Gradient Descent

Convergence Analysis of Accelerated Stochastic Gradient Descent under the Growth Condition

Faster Asynchronous SGD

Asynchronous Stochastic Gradient Descent with Decoupled Backpropagation and Layer-Wise Updates

Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigms

Enhancing Stochastic Gradient Descent: A Unified Framework and Novel Acceleration Methods for Faster Convergence

On the Convergence Properties of a K-step Averaging Stochastic Gradient Descent Algorithm for Nonconvex Optimization

A Continuous-Time Analysis of Distributed Stochastic Gradient

Asynchronous Decentralized Accelerated Stochastic Gradient Descent

Distributed Stochastic Gradient Descent with Staleness: A Stochastic Delay Differential Equation Based Framework