Abstract:We present a formal measure-theoretical theory of neural networks (NN) built on probability coupling theory. Our main contributions are summarized as follows. * Built on the formalism of probability coupling theory, we derive an algorithm framework, named Hierarchical Measure Group and Approximate System (HMGAS), nicknamed S-System, that is designed to learn the complex hierarchical, statistical dependency in the physical world. * We show that NNs are special cases of S-System when the probability kernels assume certain exponential family distributions. Activation Functions are derived formally. We further endow geometry on NNs through information geometry, show that intermediate feature spaces of NNs are stochastic manifolds, and prove that "distance" between samples is contracted as layers stack up. * S-System shows NNs are inherently stochastic, and under a set of realistic boundedness and diversity conditions, it enables us to prove that for large size nonlinear deep NNs with a class of losses, including the hinge loss, all local minima are global minima with zero loss errors, and regions around the minima are flat basins where all eigenvalues of Hessians are concentrated around zero, using tools and ideas from mean field theory, random matrix theory, and nonlinear operator equations. * S-System, the information-geometry structure and the optimization behaviors combined completes the analog between Renormalization Group (RG) and NNs. It shows that a NN is a complex adaptive system that estimates the statistic dependency of microscopic object, e.g., pixels, in multiple scales. Unlike clear-cut physical quantity produced by RG in physics, e.g., temperature, NNs renormalize/recompose manifolds emerging through learning/optimization that divide the sample space into highly semantically meaningful groups that are dictated by supervised labels (in supervised NNs).

Orders-of-coupling representation with a single neural network with optimal neuron activation functions and without nonlinear parameter optimization

Discovering the Representation Bottleneck of Graph Neural Networks from Multi-order Interactions

Neural network with optimal neuron activation functions based on additive Gaussian process regression

Effective Many-body Interactions in Reduced-Dimensionality Spaces Through Neural Network Models

Linearization Turns Neural Operators into Function-Valued Gaussian Processes

Formation of Representations in Neural Networks

Neural network representation of quantum systems

Function-Space Optimality of Neural Architectures With Multivariate Nonlinearities

Simulating Quantum Many-Body States with Neural-Network Exponential Ansatz

Probabilistic graphs using coupled random variables

Quantum optical neural networks with programmable nonlinearities

Measure, Manifold, Learning, and Optimization: A Theory Of Neural Networks

Neural network solutions of bosonic quantum systems in one dimension

Constructing exact representations of quantum many-body systems with deep neural networks

Symmetries and many-body excited states with neural-network quantum states

Transformer neural networks and quantum simulators: a hybrid approach for simulating strongly correlated systems

Solving the Quantum Many-Body Problem with Artificial Neural Networks

A simple linear algebra identity to optimize Large-Scale Neural Network Quantum States

Linking Gaussian process regression with data-driven manifold embeddings for nonlinear data fusion

Empowering deep neural quantum states through efficient optimization

Approximating quantum many-body wave-functions using artificial neural networks