Abstract:The recently discovered Neural Collapse (NC) phenomenon occurs pervasively in today's deep net training paradigm of driving cross-entropy (CE) loss towards zero. During NC, last-layer features collapse to their class-means, both classifiers and class-means collapse to the same Simplex Equiangular Tight Frame, and classifier behavior collapses to the nearest-class-mean decision rule. Recent works demonstrated that deep nets trained with mean squared error (MSE) loss perform comparably to those trained with CE. As a preliminary, we empirically establish that NC emerges in such MSE-trained deep nets as well through experiments on three canonical networks and five benchmark datasets. We provide, in a Google Colab notebook, PyTorch code for reproducing MSE-NC and CE-NC: at <a class="link-external link-https" href="https://colab.research.google.com/github/neuralcollapse/neuralcollapse/blob/main/neuralcollapse.ipynb" rel="external noopener nofollow">this https URL</a>. The analytically-tractable MSE loss offers more mathematical opportunities than the hard-to-analyze CE loss, inspiring us to leverage MSE loss towards the theoretical investigation of NC. We develop three main contributions: (I) We show a new decomposition of the MSE loss into (A) terms directly interpretable through the lens of NC and which assume the last-layer classifier is exactly the least-squares classifier; and (B) a term capturing the deviation from this least-squares classifier. (II) We exhibit experiments on canonical datasets and networks demonstrating that term-(B) is negligible during training. This motivates us to introduce a new theoretical construct: the central path, where the linear classifier stays MSE-optimal for feature activations throughout the dynamics. (III) By studying renormalized gradient flow along the central path, we derive exact dynamics that predict NC.

Prevalence of Neural Collapse during the terminal phase of deep learning training

Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networks

Perturbation Analysis of Neural Collapse

Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse

Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path

An Unconstrained Layer-Peeled Perspective on Neural Collapse

The Prevalence of Neural Collapse in Neural Multivariate Regression

Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?

Neural (Tangent Kernel) Collapse

Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decay

Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data

Limitations of Neural Collapse for Understanding Generalization in Deep Learning

Progressive Feedforward Collapse of ResNet Training

Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data

All-around Neural Collapse for Imbalanced Classification

Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?

Kernel vs. Kernel: Exploring How the Data Structure Affects Neural Collapse

Towards understanding neural collapse in supervised contrastive learning with the information bottleneck method

Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model

Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representations

The Persistence of Neural Collapse Despite Low-Rank Bias: An Analytic Perspective Through Unconstrained Features