Abstract:The Neural Network Field Theory correspondence (NNFT) is a mapping from neural network (NN) architectures into the space of statistical field theories (SFTs). The Bayesian renormalization group (BRG) is an information-theoretic coarse graining scheme that generalizes the principles of the Exact Renormalization Group (ERG) to arbitrarily parameterized probability distributions, including those of NNs. In BRG, coarse graining is performed in parameter space with respect to an information-theoretic distinguishability scale set by the Fisher information metric. In this paper, we unify NNFT and BRG to form a powerful new framework for exploring the space of NNs and SFTs, which we coin BRG-NNFT. With BRG-NNFT, NN training dynamics can be interpreted as inducing a flow in the space of SFTs from the information-theoretic `IR' $\rightarrow$ `UV'. Conversely, applying an information-shell coarse graining to the trained network's parameters induces a flow in the space of SFTs from the information-theoretic `UV' $\rightarrow$ `IR'. When the information-theoretic cutoff scale coincides with a standard momentum scale, BRG is equivalent to ERG. We demonstrate the BRG-NNFT correspondence on two analytically tractable examples. First, we construct BRG flows for trained, infinite-width NNs, of arbitrary depth, with generic activation functions. As a special case, we then restrict to architectures with a single infinitely-wide layer, scalar outputs, and generalized cos-net activations. In this case, we show that BRG coarse-graining corresponds exactly to the momentum-shell ERG flow of a free scalar SFT. Our analytic results are corroborated by a numerical experiment in which an ensemble of asymptotically wide NNs are trained and subsequently renormalized using an information-shell BRG scheme.

Scale-invariant feature extraction of neural network and renormalization group flow

Is Deep Learning a Renormalization Group Flow?

Interpreting Deep Learning by Establishing a Rigorous Corresponding Relationship with Renormalization Group

Bayesian RG Flow in Neural Network Field Theories

A Deep Dive into the Connections Between the Renormalization Group and Deep Learning in the Ising Model

Extracting Critical Exponents by Finite-Size Scaling with Convolutional Neural Networks

RG-Flow: A hierarchical and explainable flow model based on renormalization group and sparse prior

Frequency and Scale Perspectives of Feature Extraction

Riesz Networks: Scale-Invariant Neural Networks in a Single Forward Pass

Renormalization Group-Motivated Learning

Deep learning and the renormalization group

An exact mapping between the Variational Renormalization Group and Deep Learning

Mutual information, neural networks and the renormalization group

Neural Network Renormalization Group

Renormalization group flow and fixed-point in tensor network representations

Riesz networks: scale invariant neural networks in a single forward pass

Scaling ResNets in the Large-depth Regime

A Scale-Invariant Diagnostic Approach Towards Understanding Dynamics of Deep Neural Networks

FFEINR: Flow Feature-Enhanced Implicit Neural Representation for Spatio-temporal Super-Resolution

Scaling properties of scale-free networks in degree-thresholding renormalization flows

FFEINR: flow feature-enhanced implicit neural representation for spatiotemporal super-resolution