Abstract:This paper investigates asymptotic behaviors of gradient descent algorithms (particularly accelerated gradient descent and stochastic gradient descent) in the context of stochastic optimization arising in statistics and machine learning where objective functions are estimated from available data. We show that these algorithms can be computationally modeled by continuous-time ordinary or stochastic differential equations. We establish gradient flow central limit theorems to describe the limiting dynamic behaviors of these computational algorithms and the large-sample performances of the related statistical procedures, as the number of algorithm iterations and data size both go to infinity, where the gradient flow central limit theorems are governed by some linear ordinary or stochastic differential equations like time-dependent Ornstein-Uhlenbeck processes. We illustrate that our study can provide a novel unified framework for a joint computational and statistical asymptotic analysis, where the computational asymptotic analysis studies dynamic behaviors of these algorithms with the time (or the number of iterations in the algorithms), the statistical asymptotic analysis investigates large sample behaviors of the statistical procedures (like estimators and classifiers) that the algorithms are applied to compute, and in fact the statistical procedures are equal to the limits of the random sequences generated from these iterative algorithms as the number of iterations goes to infinity. The joint analysis results based on the obtained gradient flow central limit theorems can identify four factors - learning rate, batch size, gradient covariance, and Hessian - to derive new theory regarding the local minima found by stochastic gradient descent for solving non-convex optimization problems.

A theory of generalised coordinates for stochastic differential equations

Lévy areas, Wong Zakai anomalies in diffusive limits of Deterministic Lagrangian Multi-Time Dynamics

Identifying Drift, Diffusion, and Causal Structure from Temporal Snapshots

An improved characterisation of regular generalised functions of white noise and an application to singular SPDEs

Generalised shot noise representations of stochastic systems driven by non-Gaussian Lévy processes

Generalised shot-noise representations of stochastic systems driven by non-Gaussian Lévy processes

Langevin formulation of a subdiffusive continuous time random walk in physical time

Dynamics of Stochastic Reaction-Diffusion Equations

Invariance Principles for [math]-Brownian-Motion-Driven Stochastic Differential Equations and Their Applications to [math]-Stochastic Control

A Primer on Stochastic Partial Differential Equations with Spatially Correlated Noise

A regularised Dean-Kawasaki model: derivation and analysis

Stochastic PDE approach to fluctuating interfaces

Numerics with coordinate transforms for efficient Brownian dynamics simulations

Numerical Methods with Coordinate Transforms for Efficient Brownian Dynamics Simulations

Langevin formulation of a subdiffusive continuous-time random walk in physical time

Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index models

Density fluctuations in weakly interacting particle systems via the Dean-Kawasaki equation

Stochastic Transport with Lévy Noise -- Fully Discrete Numerical Approximation

Fluctuations in Wasserstein dynamics on Graphs

Interpretation of generalized Langevin equations

Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigms