Abstract:Representations of the world environment play a crucial role in artificial intelligence. It is often inefficient to conduct reasoning and inference directly in the space of raw sensory representations, such as pixel values of images. Representation learning allows us to automatically discover suitable representations from raw sensory data. For example, given raw sensory data, a deep neural network learns nonlinear representations at its hidden layers, which are subsequently used for classification (or regression) at its output layer. This happens implicitly during training through minimizing a supervised or unsupervised loss in common practical regimes of deep learning, unlike the neural tangent kernel (NTK) regime. In this paper, we study the dynamics of such implicit nonlinear representation learning, which is beyond the NTK regime. We identify a pair of a new assumption and a novel condition, called the common model structure assumption and the data-architecture alignment condition. Under the common model structure assumption, the data-architecture alignment condition is shown to be sufficient for the global convergence and necessary for the global optimality. Moreover, our theory explains how and when increasing the network size does and does not improve the training behaviors in the practical regime. Our results provide practical guidance for designing a model structure: e.g., the common model structure assumption can be used as a justification for using a particular model structure instead of others. We also derive a new training framework based on the theory. The proposed framework is empirically shown to maintain competitive (practical) test performances while providing global convergence guarantees for deep residual neural networks with convolutions, skip connections, and batch normalization with standard benchmark datasets, including CIFAR-10, CIFAR-100, and SVHN.

Low-dimensional Intrinsic Dimension Reveals a Phase Transition in Gradient-Based Learning of Deep Neural Networks

Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks

Universal Scaling Laws of Absorbing Phase Transitions in Artificial Deep Neural Networks

Learning in PINNs: Phase transition, total diffusion, and generalization

Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionality

Grokking as a First Order Phase Transition in Two Layer Networks

A spring-block theory of feature learning in deep neural networks

Effective Rank and the Staircase Phenomenon: New Insights into Neural Network Training Dynamics

Dynamics of Deep Neural Networks and Neural Tangent Hierarchy

How many degrees of freedom do we need to train deep networks: a loss landscape perspective

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Learning Time-Scales in Two-Layers Neural Networks

Unifying Low Dimensional Observations in Deep Learning Through the Deep Linear Unconstrained Feature Model

The Law of Parsimony in Gradient Descent for Learning Deep Linear Networks

Multi-scale Feature Learning Dynamics: Insights for Double Descent

Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations

Disentangling feature and lazy training in deep neural networks

Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functions

Understanding Dynamics of Nonlinear Representation Learning and Its Application