Abstract:This paper investigates asymptotic behaviors of gradient descent algorithms (particularly accelerated gradient descent and stochastic gradient descent) in the context of stochastic optimization arising in statistics and machine learning where objective functions are estimated from available data. We show that these algorithms can be computationally modeled by continuous-time ordinary or stochastic differential equations. We establish gradient flow central limit theorems to describe the limiting dynamic behaviors of these computational algorithms and the large-sample performances of the related statistical procedures, as the number of algorithm iterations and data size both go to infinity, where the gradient flow central limit theorems are governed by some linear ordinary or stochastic differential equations like time-dependent Ornstein-Uhlenbeck processes. We illustrate that our study can provide a novel unified framework for a joint computational and statistical asymptotic analysis, where the computational asymptotic analysis studies dynamic behaviors of these algorithms with the time (or the number of iterations in the algorithms), the statistical asymptotic analysis investigates large sample behaviors of the statistical procedures (like estimators and classifiers) that the algorithms are applied to compute, and in fact the statistical procedures are equal to the limits of the random sequences generated from these iterative algorithms as the number of iterations goes to infinity. The joint analysis results based on the obtained gradient flow central limit theorems can identify four factors - learning rate, batch size, gradient covariance, and Hessian - to derive new theory regarding the local minima found by stochastic gradient descent for solving non-convex optimization problems.

Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithm

Extremely Fast Convergence Rates for Extremum Seeking Control with Polyak-Ruppert Averaging

Zeroth-order Asynchronous Doubly Stochastic Algorithm with Variance Reduction.

Stochastic Zeroth-order Optimization Via Variance Reduction Method.

Accelerated zero-order SGD under high-order smoothness and overparameterized regime

Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming

Variance-Reduced Gradient Estimator for Nonconvex Zeroth-Order Distributed Optimization

Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigms

Resampling Stochastic Gradient Descent Cheaply for Efficient Uncertainty Quantification

Scalable Statistical Inference in Non-parametric Least Squares

Online estimation of the asymptotic variance for averaged stochastic gradient algorithms

Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm

Stochastic Zeroth Order Gradient and Hessian Estimators: Variance Reduction and Refined Bias Bounds

The "Black-Box" Optimization Problem: Zero-Order Accelerated Stochastic Method via Kernel Approximation

Nonasymptotic Analysis of Stochastic Gradient Descent with the Richardson-Romberg Extrapolation

The Black-Box Optimization Problem: Zero-Order Accelerated Stochastic Method via Kernel Approximation

Zeroth-order algorithms for stochastic distributed nonconvex optimization

Gradient-free Stein Variational Gradient Descent with Kernel Approximation

Online Inference with Debiased Stochastic Gradient Descent

Small errors in random zeroth-order optimization are imaginary

Finite-Time High-Probability Bounds for Polyak–Ruppert Averaged Iterates of Linear Stochastic Approximation