Abstract:We discover restrained numerical instabilities in current training practices of deep networks with stochastic gradient descent (SGD), and its variants. We show numerical error (on the order of the smallest floating point bit and thus the most extreme or limiting numerical perturbations induced from floating point arithmetic in training deep nets can be amplified significantly and result in significant test accuracy variance (sensitivities), comparable to the test accuracy variance due to stochasticity in SGD. We show how this is likely traced to instabilities of the optimization dynamics that are restrained, i.e., localized over iterations and regions of the weight tensor space. We do this by presenting a theoretical framework using numerical analysis of partial differential equations (PDE), and analyzing the gradient descent PDE of convolutional neural networks (CNNs). We show that it is stable only under certain conditions on the learning rate and weight decay. We show that rather than blowing up when the conditions are violated, the instability can be restrained. We show this is a consequence of the non-linear PDE associated with the gradient descent of the CNN, whose local linearization changes when over-driving the step size of the discretization, resulting in a stabilizing effect. We link restrained instabilities to the recently discovered Edge of Stability (EoS) phenomena, in which the stable step size predicted by classical theory is exceeded while continuing to optimize the loss and still converging. Because restrained instabilities occur at the EoS, our theory provides new insights and predictions about the EoS, in particular, the role of regularization and the dependence on the network complexity.

Exponential Decay of Sensitivity in Graph-Structured Nonlinear Programs

Exponential Decay in the Sensitivity Analysis of Nonlinear Dynamic Programming

Solving Nonsmooth Nonconvex Compound Stochastic Programs with Applications to Risk Measure Minimization

Distributed Sequential Quadratic Programming with Overlapping Graph Decomposition and Exact Augmented Lagrangian

Sensitivity Analysis for Piecewise-Affine Approximations of Nonlinear Programs with Polytopic Constraints

Numerical method for nonlinear Kolmogorov PDEs via sensitivity analysis

Sharp Decay Estimates in Local Sensitivity Analysis for Evolution Equations with Uncertainties: from ODEs to Linear Kinetic Equations

Convergence of successive linear programming algorithms for noisy functions

An explicit analysis of the entropic penalty in linear programming

Nonlinear Perturbation-based Non-Convex Optimization over Time-Varying Networks

Super Level Sets and Exponential Decay: A Synergistic Approach to Stable Neural Network Training

Multilinear Extensions in Submodular Optimization for Optimal Sensor Scheduling in Nonlinear Networks

Asymptotic behavior and decay estimates of the solutions for a nonlinear parabolic equation with exponential nonlinearity

Sensitivity of matrix function based network communicability measures: Computational methods and a priori bounds

Solving Nonsmooth and Nonconvex Compound Stochastic Programs with Applications to Risk Measure Minimization.

Exponential Stability of Slowly Decaying Solutions to the Kinetic-Fokker-Planck Equation

A PDE-based Explanation of Extreme Numerical Sensitivities and Edge of Stability in Training Neural Networks

On the connections between optimization algorithms, Lyapunov functions, and differential equations: theory and insights

Linear Convergence of Stochastic Iterative Greedy Algorithms With Sparse Constraints

Perturbations in PDE-constrained optimal control decay exponentially in space

Graphon Particle Systems, Part II: Dynamics of Distributed Stochastic Continuum Optimization