Abstract:In the past decade, significant strides in deep learning have led to numerous groundbreaking applications. Despite these advancements, the understanding of the high generalizability of deep learning, especially in such an over-parametrized space, remains limited. For instance, in deep neural networks (DNNs), their internal representations, decision-making mechanism, absence of overfitting in an over-parametrized space, superior generalizability, etc., remain less understood. Successful applications are often considered as empirical rather than scientific achievement. This paper delves into the loss landscape of DNNs through the lens of spin glass in statistical physics, a system characterized by a complex energy landscape with numerous metastable states, as a novel perspective in understanding how DNNs work. We investigated the loss landscape of single hidden layer neural networks activated by Rectified Linear Unit (ReLU) function, and introduced several protocols to examine the analogy between DNNs and spin glass. Specifically, we used (1) random walk in the parameter space of DNNs to unravel the structures in their loss landscape; (2) a permutation-interpolation protocol to study the connection between copies of identical regions in the loss landscape due to the permutation symmetry in the hidden layers; (3) hierarchical clustering to reveal the hierarchy among trained solutions of DNNs, reminiscent of the so-called Replica Symmetry Breaking (RSB) phenomenon (i.e. the Parisi solution) in spin glass; (4) finally, we examine the relationship between the ruggedness of DNN's loss landscape and its generalizability, showing an improvement of flattened minima.

Reply to: Deep reinforced learning heuristic tested on spin-glass ground states: The larger picture

Reply to: Deep reinforced learning heuristic tested on spin-glass ground states: The larger picture

Finding Ground States of Sherrington-Kirkpatrick Spin Glass by Modified Extremal Optimization

Searching for spin glass ground states through deep reinforcement learning

Learning Spin Liquids on a Honeycomb Lattice with Artificial Neural Networks.

Finding the ground state of spin Hamiltonians with reinforcement learning

Short-sighted deep learning

Deep Learning Illuminates Spin and Lattice Interaction in Magnetic Materials

Self-Supervised Learning of Generative Spin-Glasses with Normalizing Flows

Deep Learning the Ising Model Near Criticality

Spin glasses and algorithm benchmarks: A one-dimensional view

Zero-temperature Monte Carlo simulations of two-dimensional quantum spin glasses guided by neural network states

Spin glass theory and its new challenge: structured disorder

Nature of spin glass order in physical dimensions

How to compute efficiently the analytical solution to Heisenberg spin glass models on sparse random graphs and their de Almeida-Thouless line

Exploring Loss Landscapes through the Lens of Spin Glass Theory

Rademacher complexity and spin glasses: A link between the replica and statistical theories of learning

Real Spin Glasses Relax Slowly in the Shade of Hierarchical Trees

Machine learning force-field models for metallic spin glass

Central Limit Theorem of Overlap for the Mean Field Ghatak-Sherrington model

Complete Realization of Energy Landscape and Non-equilibrium Trapping Dynamics in Spin Glass and Optimization Problem