Abstract:In this work, we reveal a strong implicit bias of stochastic gradient descent (SGD) that drives overly expressive networks to much simpler subnetworks, thereby dramatically reducing the number of independent parameters, and improving generalization. To reveal this bias, we identify invariant sets, or subsets of parameter space that remain unmodified by SGD. We focus on two classes of invariant sets that correspond to simpler (sparse or low-rank) subnetworks and commonly appear in modern architectures. Our analysis uncovers that SGD exhibits a property of stochastic attractivity towards these simpler invariant sets. We establish a sufficient condition for stochastic attractivity based on a competition between the loss landscape's curvature around the invariant set and the noise introduced by stochastic gradients. Remarkably, we find that an increased level of noise strengthens attractivity, leading to the emergence of attractive invariant sets associated with saddle-points or local maxima of the train loss. We observe empirically the existence of attractive invariant sets in trained deep neural networks, implying that SGD dynamics often collapses to simple subnetworks with either vanishing or redundant neurons. We further demonstrate how this simplifying process of stochastic collapse benefits generalization in a linear teacher-student framework. Finally, through this analysis, we mechanistically explain why early training with large learning rates for extended periods benefits subsequent generalization.

Why Does Multi-Epoch Training Help?

Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passes

Optimal Adaptive and Accelerated Stochastic Gradient Descent

Demystifying SGD with Doubly Stochastic Gradients

An Alternative View: When Does SGD Escape Local Minima?

Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions

Improved Learning Rates for Stochastic Optimization: Two Theoretical Viewpoints

Accelerated Gradient-free Neural Network Training by Multi-convex Alternating Optimization

The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents

The Optimality of (Accelerated) SGD for High-Dimensional Quadratic Optimization

On the Generalization of Stochastic Gradient Descent with Momentum

Dynamic of Stochastic Gradient Descent with State-Dependent Noise

Stochastic normalized gradient descent with momentum for large-batch training

Asynchronous Accelerated Stochastic Gradient Descent.

Stochastic Gradient Descent with Biased but Consistent Gradient Estimators

Gradient Descent Optimization in Deep Learning Model Training Based on Multistage and Method Combination Strategy

Risk Bounds of Accelerated SGD for Overparameterized Linear Regression

Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks

Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization

Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization

OD-SGD: One-Step Delay Stochastic Gradient Descent for Distributed Training.