Abstract:We explore a one-to-one correspondence between a neural network (NN) and a statistical mechanical spin model where neurons are mapped to Ising spins and weights to spin-spin couplings. The process of training an NN produces a family of spin Hamiltonians parameterized by training time. We study the magnetic phases and the melting transition temperature as training progresses. First, we prove analytically that the common initial state before training--an NN with independent random weights--maps to a layered version of the classical Sherrington-Kirkpatrick spin glass exhibiting a replica symmetry breaking. The spin-glass-to-paramagnet transition temperature is calculated. Further, we use the Thouless-Anderson-Palmer (TAP) equations--a theoretical technique to analyze the landscape of energy minima of random systems--to determine the evolution of the magnetic phases on two types of NNs (one with continuous and one with binarized activations) trained on the MNIST dataset. The two NN types give rise to similar results, showing a quick destruction of the spin glass and the appearance of a phase with a hidden order, whose melting transition temperature $T_c$ grows as a power law in training time. We also discuss the properties of the spectrum of the spin system's bond matrix in the context of rich vs. lazy learning. We suggest that this statistical mechanical view of NNs provides a useful unifying perspective on the training process, which can be viewed as selecting and strengthening a symmetry-broken state associated with the training task.

Graded Symmetry Algebras of Time-Dependent Evolution Equations and Application to the Modified KP equations

TAP free energy, spin glasses, and variational inference

The thermodynamic limit in mean field neural networks

Statistical Mechanics of a Hopfield Neural-Network Model in a Transverse Field

Out-of-equilibrium dynamics of the Hopfield model in its spin-glass phase

Neural Networks as Spin Models: From Glass to Hidden Order Through Training

The Slln For The Free-Energy Of A Class Of Neural Networks

On the TAP Equations via the Cavity Approach in the Generic Mixed p-Spin Models

On the TAP equations via the cavity approach in the generic mixed $p$-spin models

Mean-Field Equations for Spin Models with Orthogonal Interaction Matrices

Statistical Mechanics of a Multiconnected Hopfield Neural-Network Model in a Transverse Field

Optimizing Mean Field Spin Glasses with External Field

A Friendly Tutorial on Mean-Field Spin Glass Techniques for Non-Physicists

Phase diagram and large deviations in the free-energy of mean-field spin-glasses

Free energy equivalence between mean-field models and nonsparsely diluted mean-field models

Mean-field variational inference with the TAP free energy: Geometric and statistical properties in linear models

A new mean field approach to finite spin systems

Mean-field theory is exact for Ising spin glass models with Kac potential in non-additive limit on Nishimori line

Hopfield Spin-Glass Model in a Transverse Field.

The infinite-range quantum transverse Ising spin-glass model

Zero-temperature Monte Carlo simulations of two-dimensional quantum spin glasses guided by neural network states