Abstract:This paper investigates asymptotic behaviors of gradient descent algorithms (particularly accelerated gradient descent and stochastic gradient descent) in the context of stochastic optimization arising in statistics and machine learning where objective functions are estimated from available data. We show that these algorithms can be computationally modeled by continuous-time ordinary or stochastic differential equations. We establish gradient flow central limit theorems to describe the limiting dynamic behaviors of these computational algorithms and the large-sample performances of the related statistical procedures, as the number of algorithm iterations and data size both go to infinity, where the gradient flow central limit theorems are governed by some linear ordinary or stochastic differential equations like time-dependent Ornstein-Uhlenbeck processes. We illustrate that our study can provide a novel unified framework for a joint computational and statistical asymptotic analysis, where the computational asymptotic analysis studies dynamic behaviors of these algorithms with the time (or the number of iterations in the algorithms), the statistical asymptotic analysis investigates large sample behaviors of the statistical procedures (like estimators and classifiers) that the algorithms are applied to compute, and in fact the statistical procedures are equal to the limits of the random sequences generated from these iterative algorithms as the number of iterations goes to infinity. The joint analysis results based on the obtained gradient flow central limit theorems can identify four factors - learning rate, batch size, gradient covariance, and Hessian - to derive new theory regarding the local minima found by stochastic gradient descent for solving non-convex optimization problems.

Thresholds of descending algorithms in inference problems

Statistical physics of inference: Thresholds and algorithms

High-dimensional non-convex landscapes and gradient descent dynamics

Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

Disordered Systems Insights on Computational Hardness

Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigms

Statistical physics of complex systems: glasses, spin glasses, continuous constraint satisfaction problems, high-dimensional inference and neural networks

Inverse statistical problems: from the inverse Ising problem to data science

Comparing Dynamics: Deep Neural Networks versus Glassy Systems

Bounds on the rates of statistical divergences and mutual information via stochastic thermodynamics

Statistical algorithms for low-frequency diffusion data: A PDE approach

Quantum gradient descent algorithms for nonequilibrium steady states and linear algebraic systems

Limit Theorems for Stochastic Gradient Descent with Infinite Variance

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Beyond the Edge of Stability via Two-step Gradient Updates

The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects

On the Overlooked Structure of Stochastic Gradients

Fisher-Rao Gradient Flow: Geodesic Convexity and Functional Inequalities