Abstract:Neural collapse (NC) is a phenomenon that emerges at the terminal phase of the training (TPT) of deep neural networks (DNNs). The features of the data in the same class collapse to their respective sample means and the sample means exhibit a simplex equiangular tight frame (ETF). In the past few years, there has been a surge of works that focus on explaining why the NC occurs and how it affects generalization. Since the DNNs are notoriously difficult to analyze, most works mainly focus on the unconstrained feature model (UFM). While the UFM explains the NC to some extent, it fails to provide a complete picture of how the network architecture and the dataset affect NC. In this work, we focus on shallow ReLU neural networks and try to understand how the width, depth, data dimension, and statistical property of the training dataset influence the neural collapse. We provide a complete characterization of when the NC occurs for two or three-layer neural networks. For two-layer ReLU neural networks, a sufficient condition on when the global minimizer of the regularized empirical risk function exhibits the NC configuration depends on the data dimension, sample size, and the signal-to-noise ratio in the data instead of the network width. For three-layer neural networks, we show that the NC occurs as long as the first layer is sufficiently wide. Regarding the connection between NC and generalization, we show the generalization heavily depends on the SNR (signal-to-noise ratio) in the data: even if the NC occurs, the generalization can still be bad provided that the SNR in the data is too low. Our results significantly extend the state-of-the-art theoretical analysis of the N C under the UFM by characterizing the emergence of the N C under shallow nonlinear networks and showing how it depends on data properties and network architecture.

Towards understanding neural collapse in supervised contrastive learning with the information bottleneck method

The Exploration of Neural Collapse under Imbalanced Data

Limitations of Neural Collapse for Understanding Generalization in Deep Learning

An Unconstrained Layer-Peeled Perspective on Neural Collapse

Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?

Prevalence of Neural Collapse during the terminal phase of deep learning training

Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representations

Perturbation Analysis of Neural Collapse

Progressive Feedforward Collapse of ResNet Training

Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse

Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data

Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data

The Persistence of Neural Collapse Despite Low-Rank Bias: An Analytic Perspective Through Unconstrained Features

Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decay

Generalizing and Decoupling Neural Collapse Via Hyperspherical Uniformity Gap

Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?

Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Model

On the Robustness of Neural Collapse and the Neural Collapse of Robustness

The Prevalence of Neural Collapse in Neural Multivariate Regression

Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networks

Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP)