Abstract:We present a formal measure-theoretical theory of neural networks (NN) built on probability coupling theory. Our main contributions are summarized as follows. * Built on the formalism of probability coupling theory, we derive an algorithm framework, named Hierarchical Measure Group and Approximate System (HMGAS), nicknamed S-System, that is designed to learn the complex hierarchical, statistical dependency in the physical world. * We show that NNs are special cases of S-System when the probability kernels assume certain exponential family distributions. Activation Functions are derived formally. We further endow geometry on NNs through information geometry, show that intermediate feature spaces of NNs are stochastic manifolds, and prove that "distance" between samples is contracted as layers stack up. * S-System shows NNs are inherently stochastic, and under a set of realistic boundedness and diversity conditions, it enables us to prove that for large size nonlinear deep NNs with a class of losses, including the hinge loss, all local minima are global minima with zero loss errors, and regions around the minima are flat basins where all eigenvalues of Hessians are concentrated around zero, using tools and ideas from mean field theory, random matrix theory, and nonlinear operator equations. * S-System, the information-geometry structure and the optimization behaviors combined completes the analog between Renormalization Group (RG) and NNs. It shows that a NN is a complex adaptive system that estimates the statistic dependency of microscopic object, e.g., pixels, in multiple scales. Unlike clear-cut physical quantity produced by RG in physics, e.g., temperature, NNs renormalize/recompose manifolds emerging through learning/optimization that divide the sample space into highly semantically meaningful groups that are dictated by supervised labels (in supervised NNs).

Neural Networks: Further Insights into Error Function, Generalized Weights and Others

A Gentle Introduction to Artificial Neural Networks

Neural Networks: Computational Models and Applications - ReadingSample

Opening the Black Box of Neural Networks: Methods for Interpreting Neural Network Models in Clinical Applications

Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theories

A Novel Method to Compute the Weights of Neural Networks

Measure, Manifold, Learning, and Optimization: A Theory Of Neural Networks

Activation Adaptation in Neural Networks

Neural Redshift: Random Networks are not Random Functions

A Fine-Grained Spectral Perspective on Neural Networks

A Gegenbauer Neural Network with Regularized Weights Direct Determination for Classification

How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teachers

On the Complexity of Learning Neural Networks

Understanding Artificial Neural Network's Behavior from Neuron Activation Perspective

Towards Understanding the Generalization of Graph Neural Networks.

Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selection

Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss

Representations and generalization in artificial and brain neural networks

Towards Understanding Generalization of Graph Neural Networks.

Explore the Knowledge contained in Network Weights to Obtain Sparse Neural Networks

Feature Learning and Generalization in Deep Networks with Orthogonal Weights