Abstract:This paper investigates asymptotic behaviors of gradient descent algorithms (particularly accelerated gradient descent and stochastic gradient descent) in the context of stochastic optimization arising in statistics and machine learning where objective functions are estimated from available data. We show that these algorithms can be computationally modeled by continuous-time ordinary or stochastic differential equations. We establish gradient flow central limit theorems to describe the limiting dynamic behaviors of these computational algorithms and the large-sample performances of the related statistical procedures, as the number of algorithm iterations and data size both go to infinity, where the gradient flow central limit theorems are governed by some linear ordinary or stochastic differential equations like time-dependent Ornstein-Uhlenbeck processes. We illustrate that our study can provide a novel unified framework for a joint computational and statistical asymptotic analysis, where the computational asymptotic analysis studies dynamic behaviors of these algorithms with the time (or the number of iterations in the algorithms), the statistical asymptotic analysis investigates large sample behaviors of the statistical procedures (like estimators and classifiers) that the algorithms are applied to compute, and in fact the statistical procedures are equal to the limits of the random sequences generated from these iterative algorithms as the number of iterations goes to infinity. The joint analysis results based on the obtained gradient flow central limit theorems can identify four factors - learning rate, batch size, gradient covariance, and Hessian - to derive new theory regarding the local minima found by stochastic gradient descent for solving non-convex optimization problems.

On Markov Chain Gradient Descent

Decentralized Markov Chain Gradient Descent

On the Decentralized Stochastic Gradient Descent With Markov Chain Sampling

Convergence of Markov Chains for Constant Step-size Stochastic Gradient Descent with Separable Functions

Convergence Rates of Accelerated Markov Gradient Descent with Applications in Reinforcement Learning

Novel Convergence Results of Adaptive Stochastic Gradient Descents

Stochastic Recursive Gradient Descent Ascent for Stochastic Nonconvex-Strongly-Concave Minimax Problems

Adaptive Random Walk Gradient Descent for Decentralized Optimization.

Analysis of Stochastic Gradient Descent in Continuous Time

Stochastic-Constrained Stochastic Optimization with Markovian Data

Continuous-time stochastic gradient descent for optimizing over the stationary distribution of stochastic differential equations

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Convergence and concentration properties of constant step-size SGD through Markov chains

Stability and Generalization of Decentralized Stochastic Gradient Descent

Optimal Adaptive and Accelerated Stochastic Gradient Descent

Adapting to Mixing Time in Stochastic Optimization with Markovian Data

Two Time-Scale Gradient Approximation Algorithm For Adaptive Markov Reward Processes

Convergence Analysis of Stochastic Gradient Descent with MCMC Estimators

Understanding the unstable convergence of gradient descent.

Non-convex Bayesian Learning via Stochastic Gradient Markov Chain Monte Carlo

Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigms