Abstract:Artificial neural networks have revolutionized machine learning in recent years, but a complete theoretical framework for their learning process is still lacking. Substantial progress has been made for infinitely wide networks. In this regime, two disparate theoretical frameworks have been used, in which the network's output is described using kernels: one framework is based on the Neural Tangent Kernel (NTK) which assumes linearized gradient descent dynamics, while the Neural Network Gaussian Process (NNGP) kernel assumes a Bayesian framework. However, the relation between these two frameworks has remained elusive. This work unifies these two distinct theories using a Markov proximal learning model for learning dynamics in an ensemble of randomly initialized infinitely wide deep networks. We derive an exact analytical expression for the network input-output function during and after learning, and introduce a new time-dependent Neural Dynamical Kernel (NDK) from which both NTK and NNGP kernels can be derived. We identify two learning phases characterized by different time scales: gradient-driven and diffusive learning. In the initial gradient-driven learning phase, the dynamics is dominated by deterministic gradient descent, and is described by the NTK theory. This phase is followed by the diffusive learning stage, during which the network parameters sample the solution space, ultimately approaching the equilibrium distribution corresponding to NNGP. Combined with numerical evaluations on synthetic and benchmark datasets, we provide novel insights into the different roles of initialization, regularization, and network depth, as well as phenomena such as early stopping and representational drift. This work closes the gap between the NTK and NNGP theories, providing a comprehensive framework for understanding the learning process of deep neural networks in the infinite width limit.

Dynamics of Deep Neural Networks and Neural Tangent Hierarchy

How does a kernel based on gradients of infinite-width neural networks come to be widely used: a review of the neural tangent kernel

Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization

Towards an Understanding of Residual Networks Using Neural Tangent Hierarchy (NTH)

Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?

Exact Convergence Rates of the Neural Tangent Kernel in the Large Depth Limit

When and why PINNs fail to train: A neural tangent kernel perspective

Tensor Programs II: Neural Tangent Kernel for Any Architecture

Connecting NTK and NNGP: A Unified Theoretical Framework for Neural Network Learning Dynamics in the Kernel Regime

A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks

A Revision of Neural Tangent Kernel-based Approaches for Neural Networks

Demystifying Lazy Training of Neural Networks from a Macroscopic Viewpoint

Fast Finite Width Neural Tangent Kernel

On the infinite width limit of neural networks with a standard parameterization

Disentangling feature and lazy training in deep neural networks

How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descent

On Random Kernels of Residual Architectures

On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspective

On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory

A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks

Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics