Abstract:The algorithms used to train neural networks, like stochastic gradient descent (SGD), have close parallels to natural processes that navigate a high-dimensional parameter space -- for example protein folding or evolution. Our study uses a Fokker-Planck approach, adapted from statistical physics, to explore these parallels in a single, unified framework. We focus in particular on the stationary state of the system in the long-time limit, which in conventional SGD is out of equilibrium, exhibiting persistent currents in the space of network parameters. As in its physical analogues, the current is associated with an entropy production rate for any given training trajectory. The stationary distribution of these rates obeys the integral and detailed fluctuation theorems -- nonequilibrium generalizations of the second law of thermodynamics. We validate these relations in two numerical examples, a nonlinear regression network and MNIST digit classification. While the fluctuation theorems are universal, there are other aspects of the stationary state that are highly sensitive to the training details. Surprisingly, the effective loss landscape and diffusion matrix that determine the shape of the stationary distribution vary depending on the simple choice of minibatching done with or without replacement. We can take advantage of this nonequilibrium sensitivity to engineer an equilibrium stationary state for a particular application: sampling from a posterior distribution of network weights in Bayesian machine learning. We propose a new variation of stochastic gradient Langevin dynamics (SGLD) that harnesses without replacement minibatching. In an example system where the posterior is exactly known, this SGWORLD algorithm outperforms SGLD, converging to the posterior orders of magnitude faster as a function of the learning rate.

Statistical physics and practical training of soft-committee machines

Weight decay induced phase transitions in multilayer neural networks

The committee machine: Computational to statistical gaps in learning a two-layers neural network

Mean-field methods and algorithmic perspectives for high-dimensional machine learning

Machine learning in and out of equilibrium

On-line learning dynamics of ReLU neural networks using statistical physics techniques

Fundamental operating regimes, hyper-parameter fine-tuning and glassiness: towards an interpretable replica-theory for trained restricted Boltzmann machines

Deep Learning the Ising Model Near Criticality

Statistical mechanics of continual learning: variational principle and mean-field potential

Dense Hebbian neural networks: a replica symmetric picture of unsupervised learning

Learning nonequilibrium statistical mechanics and dynamical phase transitions

Entropy, concentration, and learning: a statistical mechanics primer

A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit

Machine learning stochastic differential equations for the evolution of order parameters of classical many-body systems in and out of equilibrium

Dynamic neurons: A statistical physics approach for analyzing deep neural networks

Short-sighted deep learning

Dense Hebbian neural networks: a replica symmetric picture of supervised learning

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Phase determination with and without deep learning

Statistical mechanics for networks of real neurons

Training an Ising Machine with Equilibrium Propagation