Abstract:An antithetical concept, adaptive symmetry, to conservative symmetry in physics is proposed to understand the deep neural networks (DNNs). It characterizes the invariance of variance, where a biotic system explores different pathways of evolution with equal probability in absence of feedback signals, and complex functional structure emerges from quantitative accumulation of adaptive-symmetries breaking in response to feedback signals. Theoretically and experimentally, we characterize the optimization process of a DNN system as an extended adaptive-symmetry-breaking process. One particular finding is that a hierarchically large DNN would have a large reservoir of adaptive symmetries, and when the information capacity of the reservoir exceeds the complexity of the dataset, the system could absorb all perturbations of the examples and self-organize into a functional structure of zero training errors measured by a certain surrogate risk. More specifically, this process is characterized by a statistical-mechanical model that could be appreciated as a generalization of statistics physics to the DNN organized complex system, and characterizes regularities in higher dimensionality. The model consists of three constitutes that could be appreciated as the counterparts of Boltzmann distribution, Ising model, and conservative symmetry, respectively: (1) a stochastic definition/interpretation of DNNs that is a multilayer probabilistic graphical model, (2) a formalism of circuits that perform biological computation, (3) a circuit symmetry from which self-similarity between the microscopic and the macroscopic adaptability manifests. The model is analyzed with a method referred as the statistical assembly method that analyzes the coarse-grained behaviors (over a symmetry group) of the heterogeneous hierarchical many-body interaction in DNNs.

Unveiling the Link between Complexity and Symmetry: Statistical Asymmetry

Emergence of Symmetry in Complex Networks

From Asymmetry to Correlations

Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networks

Symmetry in Signals: A New Insight

Dynamical Complexity in an Asymmetric Bistable System Via Statistical Complexity Measures

A symmetry-based approach to species-rich ecological communities

Symmetry-based Structure Entropy of Complex Networks

Entanglement asymmetry as a probe of symmetry breaking

Optimization Leads to Symmetry

Sampling and statistical physics via symmetry

Symmetry within and between solutions

Symmetry in data mining and analysis: A unifying view based on hierarchy

Symmetry, Entropy, Diversity and (why not?) Quantum Statistics in Society

Asymptotic symmetry and group invariance for randomization

Structural and Symmetry Analysis of Discrete Dynamical Systems

Universal hard-edge statistics of non-Hermitian random matrices

Three Perspectives on Complexity $-$ Entropy, Compression, Subsymmetry

A rigorous and efficient approach to finding and quantifying symmetries in complex networks

Some Necessary Uniform Tests for Spherical Symmetry

Observability and Controllability of Nonlinear Networks: The Role of Symmetry