Abstract:An antithetical concept, adaptive symmetry, to conservative symmetry in physics is proposed to understand the deep neural networks (DNNs). It characterizes the invariance of variance, where a biotic system explores different pathways of evolution with equal probability in absence of feedback signals, and complex functional structure emerges from quantitative accumulation of adaptive-symmetries breaking in response to feedback signals. Theoretically and experimentally, we characterize the optimization process of a DNN system as an extended adaptive-symmetry-breaking process. One particular finding is that a hierarchically large DNN would have a large reservoir of adaptive symmetries, and when the information capacity of the reservoir exceeds the complexity of the dataset, the system could absorb all perturbations of the examples and self-organize into a functional structure of zero training errors measured by a certain surrogate risk. More specifically, this process is characterized by a statistical-mechanical model that could be appreciated as a generalization of statistics physics to the DNN organized complex system, and characterizes regularities in higher dimensionality. The model consists of three constitutes that could be appreciated as the counterparts of Boltzmann distribution, Ising model, and conservative symmetry, respectively: (1) a stochastic definition/interpretation of DNNs that is a multilayer probabilistic graphical model, (2) a formalism of circuits that perform biological computation, (3) a circuit symmetry from which self-similarity between the microscopic and the macroscopic adaptability manifests. The model is analyzed with a method referred as the statistical assembly method that analyzes the coarse-grained behaviors (over a symmetry group) of the heterogeneous hierarchical many-body interaction in DNNs.

A Scale-Invariant Diagnostic Approach Towards Understanding Dynamics of Deep Neural Networks

Dynamic neurons: A statistical physics approach for analyzing deep neural networks

Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks

Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networks

Beyond Geometry: Comparing the Temporal Structure of Computation in Neural Circuits with Dynamical Similarity Analysis

How neurons exploit fractal geometry to optimize their network connectivity

Leveraging the Graph Structure of Neural Network Training Dynamics

Neural Downscaling for Complex Systems: from Large-scale to Small-scale by Neural Operator

Deep Neural Networks via Complex Network Theory: a Perspective

Information Scaling Law of Deep Neural Networks

On The Potential of The Fractal Geometry and The CNNs Ability to Encode it

A Dynamical Model of Neural Scaling Laws

Understanding the Dynamics of DNNs Using Graph Modularity

Dynamic-scale graph neural network for fault detection

Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts

Multilevel Interpretability Of Artificial Neural Networks: Leveraging Framework And Methods From Neuroscience

Characterizing Learning Dynamics of Deep Neural Networks via Complex Networks

Deep learning systems as complex networks

Topological exploration of artificial neuronal network dynamics

Predicting Long-term Dynamics of Complex Networks via Identifying Skeleton in Hyperbolic Space

Multiscale Computations on Neural Networks: From the Individual Neuron Interactions to the Macroscopic-Level Analysis