Abstract:We present a formal measure-theoretical theory of neural networks (NN) built on probability coupling theory. Our main contributions are summarized as follows. * Built on the formalism of probability coupling theory, we derive an algorithm framework, named Hierarchical Measure Group and Approximate System (HMGAS), nicknamed S-System, that is designed to learn the complex hierarchical, statistical dependency in the physical world. * We show that NNs are special cases of S-System when the probability kernels assume certain exponential family distributions. Activation Functions are derived formally. We further endow geometry on NNs through information geometry, show that intermediate feature spaces of NNs are stochastic manifolds, and prove that "distance" between samples is contracted as layers stack up. * S-System shows NNs are inherently stochastic, and under a set of realistic boundedness and diversity conditions, it enables us to prove that for large size nonlinear deep NNs with a class of losses, including the hinge loss, all local minima are global minima with zero loss errors, and regions around the minima are flat basins where all eigenvalues of Hessians are concentrated around zero, using tools and ideas from mean field theory, random matrix theory, and nonlinear operator equations. * S-System, the information-geometry structure and the optimization behaviors combined completes the analog between Renormalization Group (RG) and NNs. It shows that a NN is a complex adaptive system that estimates the statistic dependency of microscopic object, e.g., pixels, in multiple scales. Unlike clear-cut physical quantity produced by RG in physics, e.g., temperature, NNs renormalize/recompose manifolds emerging through learning/optimization that divide the sample space into highly semantically meaningful groups that are dictated by supervised labels (in supervised NNs).

Neural Network Renormalization Group

Machine Learning and the Renormalization Group

Optimal Real-Space Renormalization-Group Transformations with Artificial Neural Networks

Renormalization Group Equations in Gauge Theories with Multiple U(1) Groups

Mutual information, neural networks and the renormalization group

Deep learning and the renormalization group

An exact mapping between the Variational Renormalization Group and Deep Learning

Machine Learning Renormalization Group for Statistical Physics

Wilsonian Renormalization of Neural Network Gaussian Processes

Laplacian Renormalization Group: An introduction to heterogeneous coarse-graining

Interpreting Deep Learning by Establishing a Rigorous Corresponding Relationship with Renormalization Group

Renormalization Group-Motivated Learning

Higher-order Laplacian Renormalization

A Variational Approach to Monte Carlo Renormalization Group

A Deep Dive into the Connections Between the Renormalization Group and Deep Learning in the Ising Model

Evolution of the Bogoluibov Renormalization Group

Bayesian RG Flow in Neural Network Field Theories

Laplacian renormalization group for heterogeneous networks

Scale-invariant feature extraction of neural network and renormalization group flow

Renormalization group for evolving networks

Measure, Manifold, Learning, and Optimization: A Theory Of Neural Networks