Abstract:Algorithm-dependent generalization error bounds are central to statistical learning theory. A learning algorithm may use a large hypothesis space, but the limited number of iterations controls its model capacity and generalization error. The impacts of stochastic gradient methods on generalization error for non-convex learning problems not only have important theoretical consequences, but are also critical to generalization errors of deep learning. In this paper, we study the generalization errors of Stochastic Gradient Langevin Dynamics (SGLD) with non-convex objectives. Two theories are proposed with non-asymptotic discrete-time analysis, using Stability and PAC-Bayesian results respectively. The stability-based theory obtains a bound of $O\left(\frac{1}{n}L\sqrt{\beta T_k}\right)$, where $L$ is uniform Lipschitz parameter, $\beta$ is inverse temperature, and $T_k$ is aggregated step sizes. For PAC-Bayesian theory, though the bound has a slower $O(1/\sqrt{n})$ rate, the contribution of each step is shown with an exponentially decaying factor by imposing $\ell^2$ regularization, and the uniform Lipschitz constant is also replaced by actual norms of gradients along trajectory. Our bounds have no implicit dependence on dimensions, norms or other capacity measures of parameter, which elegantly characterizes the phenomenon of "Fast Training Guarantees Generalization" in non-convex settings. This is the first algorithm-dependent result with reasonable dependence on aggregated step sizes for non-convex learning, and has important implications to statistical learning aspects of stochastic gradient methods in complicated models such as deep learning.

A sharp uniform-in-time error estimate for Stochastic Gradient Langevin Dynamics

Generalization Bounds for Stochastic Gradient Langevin Dynamics: A Unified View Via Information Leakage Analysis

Subsampling Error in Stochastic Gradient Langevin Diffusions

On Stationary-Point Hitting Time and Ergodicity of Stochastic Gradient Langevin Dynamics

Time-Independent Information-Theoretic Generalization Bounds for SGLD

Hitting Time of Stochastic Gradient Langevin Dynamics to Stationary Points: A Direct Analysis

On Uniform-in-time Diffusion Approximation for Stochastic Gradient Descent.

Langevin Dynamics: A Unified Perspective on Optimization via Lyapunov Potentials

Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms Via Diffusion Approximation

Utilising the CLT Structure in Stochastic Gradient based Sampling : Improved Analysis and Faster Algorithms

Functional Central Limit Theorem and Strong Law of Large Numbers for Stochastic Gradient Langevin Dynamics

Higher Order Generalization Error for First Order Discretization of Langevin Diffusion

Stochastic Approximate Gradient Descent via the Langevin Algorithm

Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints.

Low-Precision Stochastic Gradient Langevin Dynamics

Laplacian Smoothing Stochastic Gradient Markov Chain Monte Carlo

Adaptive Stochastic Gradient Langevin Dynamics: Taming Convergence and Saddle Point Escape Time

Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descent

Self-normalized Cramér-type Moderate Deviation of Stochastic Gradient Langevin Dynamics

Geometric ergodicity of SGLD via reflection coupling

Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization