Abstract:In this paper, we advance the understanding of neural network training dynamics by examining the intricate interplay of various factors introduced by weight parameters in the initialization process. Motivated by the foundational work of Luo et al. (J. Mach. Learn. Res., Vol. 22, Iss. 1, No. 71, pp 3327-3373), we explore the gradient descent dynamics of neural networks through the lens of macroscopic limits, where we analyze its behavior as width $m$ tends to infinity. Our study presents a unified approach with refined techniques designed for multi-layer fully connected neural networks, which can be readily extended to other neural network architectures. Our investigation reveals that gradient descent can rapidly drive deep neural networks to zero training loss, irrespective of the specific initialization schemes employed by weight parameters, provided that the initial scale of the output function $\kappa$ surpasses a certain threshold. This regime, characterized as the theta-lazy area, accentuates the predominant influence of the initial scale $\kappa$ over other factors on the training behavior of neural networks. Furthermore, our approach draws inspiration from the Neural Tangent Kernel (NTK) paradigm, and we expand its applicability. While NTK typically assumes that $\lim_{m\to\infty}\frac{\log \kappa}{\log m}=\frac{1}{2}$, and imposes each weight parameters to scale by the factor $\frac{1}{\sqrt{m}}$, in our theta-lazy regime, we discard the factor and relax the conditions to $\lim_{m\to\infty}\frac{\log \kappa}{\log m}>0$. Similar to NTK, the behavior of overparameterized neural networks within the theta-lazy regime trained by gradient descent can be effectively described by a specific kernel. Through rigorous analysis, our investigation illuminates the pivotal role of $\kappa$ in governing the training dynamics of neural networks.

Boundedness and Convergence Analysis of Weight Elimination for Cyclic Training of Neural Networks

Convergence Analysis of General Neural Networks under Almost Periodic Stimuli

On Convergence of Training Loss Without Reaching Stationary Points

Boundedness and Convergence of Mini-batch Gradient Method with Cyclic Dropconnect and Penalty

Weak and strong convergence analysis of Elman neural networks via weight decay regularization

Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspective

Analysis of Cyclic Dynamics for Networks of Linear Threshold Neurons.

Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decay

Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networks

Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse

Iterative Neural Networks with Bounded Weights

Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization

On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective

Analysis of Boundedness and Convergence of Online Gradient Method for Two-Layer Feedforward Neural Networks

Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks

Optimization and Generalization Guarantees for Weight Normalization

On the Weight Dynamics of Deep Normalized Networks

Convergence Rates of Training Deep Neural Networks Via Alternating Minimization Methods.

Convergence of Gradient Descent for Recurrent Neural Networks: A Nonasymptotic Analysis

Convergence Analysis for Learning Orthonormal Deep Linear Neural Networks

Demystifying Lazy Training of Neural Networks from a Macroscopic Viewpoint