Abstract:The recently discovered Neural Collapse (NC) phenomenon occurs pervasively in today's deep net training paradigm of driving cross-entropy (CE) loss towards zero. During NC, last-layer features collapse to their class-means, both classifiers and class-means collapse to the same Simplex Equiangular Tight Frame, and classifier behavior collapses to the nearest-class-mean decision rule. Recent works demonstrated that deep nets trained with mean squared error (MSE) loss perform comparably to those trained with CE. As a preliminary, we empirically establish that NC emerges in such MSE-trained deep nets as well through experiments on three canonical networks and five benchmark datasets. We provide, in a Google Colab notebook, PyTorch code for reproducing MSE-NC and CE-NC: at <a class="link-external link-https" href="https://colab.research.google.com/github/neuralcollapse/neuralcollapse/blob/main/neuralcollapse.ipynb" rel="external noopener nofollow">this https URL</a>. The analytically-tractable MSE loss offers more mathematical opportunities than the hard-to-analyze CE loss, inspiring us to leverage MSE loss towards the theoretical investigation of NC. We develop three main contributions: (I) We show a new decomposition of the MSE loss into (A) terms directly interpretable through the lens of NC and which assume the last-layer classifier is exactly the least-squares classifier; and (B) a term capturing the deviation from this least-squares classifier. (II) We exhibit experiments on canonical datasets and networks demonstrating that term-(B) is negligible during training. This motivates us to introduce a new theoretical construct: the central path, where the linear classifier stays MSE-optimal for feature activations throughout the dynamics. (III) By studying renormalized gradient flow along the central path, we derive exact dynamics that predict NC.

Generalizing and Decoupling Neural Collapse Via Hyperspherical Uniformity Gap

Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networks

All-around Neural Collapse for Imbalanced Classification

Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data

Limitations of Neural Collapse for Understanding Generalization in Deep Learning

An Unconstrained Layer-Peeled Perspective on Neural Collapse

A Neural Collapse Perspective on Feature Evolution in Graph Neural Networks

Neural (Tangent Kernel) Collapse

Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decay

Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data

The Prevalence of Neural Collapse in Neural Multivariate Regression

Progressive Feedforward Collapse of ResNet Training

Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path

Perturbation Analysis of Neural Collapse

Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Model

Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model

A Study of Neural Collapse Phenomenon: Grassmannian Frame, Symmetry and Generalization

Kernel vs. Kernel: Exploring How the Data Structure Affects Neural Collapse

Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse

Prevalence of Neural Collapse during the terminal phase of deep learning training

Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?