Abstract:Feature learning, or the ability of deep neural networks to automatically learn relevant features from raw data, underlies their exceptional capability to solve complex tasks. However, feature learning seems to be realized in different ways in fully-connected (FC) or convolutional architectures (CNNs). Empirical evidence shows that FC neural networks in the infinite-width limit eventually outperform their finite-width counterparts. Since the kernel that describes infinite-width networks does not evolve during training, whatever form of feature learning occurs in deep FC architectures is not very helpful in improving generalization. On the other hand, state-of-the-art architectures with convolutional layers achieve optimal performances in the finite-width regime, suggesting that an effective form of feature learning emerges in this case. In this work, we present a simple theoretical framework that provides a rationale for these differences, in one hidden layer networks. First, we show that the generalization performance of a finite-width FC network can be obtained by an infinite-width network, with a suitable choice of the Gaussian priors. Second, we derive a finite-width effective action for an architecture with one convolutional hidden layer and compare it with the result available for FC networks. Remarkably, we identify a completely different form of kernel renormalization: whereas the kernel of the FC architecture is just globally renormalized by a single scalar parameter, the CNN kernel undergoes a local renormalization, meaning that the network can select the local components that will contribute to the final prediction in a data-dependent way. This finding highlights a simple mechanism for feature learning that can take place in overparametrized shallow CNNs, but not in shallow FC architectures or in locally connected neural networks without weight sharing.

Three Mechanisms of Feature Learning in the Exact Solution of a Latent Variable Model

Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks

Low-dimensional Intrinsic Dimension Reveals a Phase Transition in Gradient-Based Learning of Deep Neural Networks

A spring-block theory of feature learning in deep neural networks

Critical feature learning in deep neural networks

Feature Learning in Infinite-Width Neural Networks

Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning

From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks

Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layers

Complexity Matters: Dynamics of Feature Learning in the Presence of Spurious Correlations

Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networks

Dynamics of finite width Kernel and prediction fluctuations in mean field neural networks *

Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks

Understanding Sparse Feature Updates in Deep Networks using Iterative Linearisation

Towards a theory of learning dynamics in deep state space models

Feature-Learning Networks Are Consistent Across Widths At Realistic Scales

Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamics

Learning High-Dimensional Distributions with Latent Neural Fokker-Planck Kernels

Deep Latent-Variable Kernel Learning

A Unified Framework for Kernelization: the Empirical Kernel Feature Space