Abstract:We explore a one-to-one correspondence between a neural network (NN) and a statistical mechanical spin model where neurons are mapped to Ising spins and weights to spin-spin couplings. The process of training an NN produces a family of spin Hamiltonians parameterized by training time. We study the magnetic phases and the melting transition temperature as training progresses. First, we prove analytically that the common initial state before training--an NN with independent random weights--maps to a layered version of the classical Sherrington-Kirkpatrick spin glass exhibiting a replica symmetry breaking. The spin-glass-to-paramagnet transition temperature is calculated. Further, we use the Thouless-Anderson-Palmer (TAP) equations--a theoretical technique to analyze the landscape of energy minima of random systems--to determine the evolution of the magnetic phases on two types of NNs (one with continuous and one with binarized activations) trained on the MNIST dataset. The two NN types give rise to similar results, showing a quick destruction of the spin glass and the appearance of a phase with a hidden order, whose melting transition temperature $T_c$ grows as a power law in training time. We also discuss the properties of the spectrum of the spin system's bond matrix in the context of rich vs. lazy learning. We suggest that this statistical mechanical view of NNs provides a useful unifying perspective on the training process, which can be viewed as selecting and strengthening a symmetry-broken state associated with the training task.

What problem does this paper attempt to address?

This paper explores the correspondence between Neural Networks (NNs) and spin models in statistical mechanics, and studies the magnetic phase transitions and changes in transition temperature of spin models under this correspondence through the training process. Specifically, the authors map neurons in neural networks to Ising spins, and weights to the coupling strength between spins. By training the neural network, a series of spin Hamiltonians parameterized by training time can be obtained. The main objectives of the paper include: 1. **Analysis of the Initial State**: Before the training begins, the weights in the neural network are randomly initialized. The authors demonstrate that the corresponding spin model at this time is a layered version of the classical Sherrington-Kirkpatrick spin glass, and calculate the transition temperature from the spin glass to the paramagnetic phase. 2. **Utilizing the Thouless-Anderson-Palmer (TAP) Equations**: To study the landscape of energy minima in a single instance, the paper uses the TAP equations to determine the changes in critical temperature as the training process progresses. 3. **Studying Two Types of Neural Networks**: One using continuous activation functions (such as ReLU), and the other using binarized activation functions. Both types of neural networks are trained on the MNIST dataset for handwritten digit recognition tasks. 4. **Exploring Magnetic Phase Changes During Training**: As training progresses, the spin glass phase is disrupted, and a new phase with hidden order emerges, whose melting transition temperature $T_c$ grows in a power-law manner with training time. 5. **Analyzing Changes in the Spectrum of the Weight Matrix**: The paper also discusses how the spectrum of the weight matrix evolves during training, and the connection of this evolution to "Rich Learning" and "Lazy Learning." In summary, the paper aims to deeply understand the essence of the neural network training process from a statistical mechanics perspective, particularly the different magnetic phases that appear during training and their transition mechanisms.

Neural Networks as Spin Models: From Glass to Hidden Order Through Training

Learning Spin Liquids on a Honeycomb Lattice with Artificial Neural Networks.

Deep neural networks for direct, featureless learning through observation: the case of 2d spin models

Graded Symmetry Algebras of Time-Dependent Evolution Equations and Application to the Modified KP equations

Learning to find order in disorder

Neural annealing and visualization of autoregressive neural networks in the Newman-Moore model

Influence of anisotropy on the study of critical behavior of spin models by machine learning methods

Do deep neural networks behave like structural glasses?

Zero-temperature Monte Carlo simulations of two-dimensional quantum spin glasses guided by neural network states

On the generalizability of artificial neural networks in spin models

Short-sighted deep learning

Demolition and Reinforcement of Memories in Spin-Glass-like Neural Networks

Asymmetric Sherrington-Kirkpatrick Model of Neural Networks with Random Neuronal Threshold.

Are spin-glass effects relevant to understanding realistic auto-associative networks?

Exploring Loss Landscapes through the Lens of Spin Glass Theory

Deep Learning the Ising Model Near Criticality

Towards Artificial Ising Spin Glasses: Thermal Ordering in Randomized Arrays of Ising-type Nanomagnets

Complex Spin Hamiltonian Represented by an Artificial Neural Network

Ground-state Phase-Space Structures of Two-Dimensional±jspin Glasses: A Network Approach

Comparing Dynamics: Deep Neural Networks versus Glassy Systems

The thermodynamic limit in mean field neural networks