Abstract:Recently it has been observed that neural networks exhibit Neural Collapse (NC) during the final stage of training for the classification problem. We empirically show that multivariate regression, as employed in imitation learning and other applications, exhibits Neural Regression Collapse (NRC), a new form of neural collapse: (NRC1) The last-layer feature vectors collapse to the subspace spanned by the $n$ principal components of the feature vectors, where $n$ is the dimension of the targets (for univariate regression, $n=1$); (NRC2) The last-layer feature vectors also collapse to the subspace spanned by the last-layer weight vectors; (NRC3) The Gram matrix for the weight vectors converges to a specific functional form that depends on the covariance matrix of the targets. After empirically establishing the prevalence of (NRC1)-(NRC3) for a variety of datasets and network architectures, we provide an explanation of these phenomena by modeling the regression task in the context of the Unconstrained Feature Model (UFM), in which the last layer feature vectors are treated as free variables when minimizing the loss function. We show that when the regularization parameters in the UFM model are strictly positive, then (NRC1)-(NRC3) also emerge as solutions in the UFM optimization problem. We also show that if the regularization parameters are equal to zero, then there is no collapse. To our knowledge, this is the first empirical and theoretical study of neural collapse in the context of regression. This extension is significant not only because it broadens the applicability of neural collapse to a new category of problems but also because it suggests that the phenomena of neural collapse could be a universal behavior in deep learning.

Why Your Neural Network is Still Singular and What You Can Do About It

The Persistence of Neural Collapse Despite Low-Rank Bias: An Analytic Perspective Through Unconstrained Features

Dynamics in Deep Classifiers Trained with the Square Loss: Normalization, Low Rank, Neural Collapse, and Generalization Bounds

Theory IIIb: Generalization in Deep Networks

Towards Understanding the Condensation of Two-layer Neural Networks at Initial Training.

On the Importance of Consistency in Training Deep Neural Networks

Understanding deep learning requires rethinking generalization

Neural Rank Collapse: Weight Decay and Small Within-Class Variability Yield Low-Rank Bias

Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks

Understanding deep learning (still) requires rethinking generalization

Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidance

The Multiscale Structure of Neural Network Loss Functions: The Effect on Optimization and Origin

(Almost) Smooth Sailing: Towards Numerical Stability of Neural Networks Through Differentiable Regularization of the Condition Number

Weight Conditioning for Smooth Optimization of Neural Networks

Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?

When Do Neural Networks Outperform Kernel Methods?

Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse

The Efficacy of Regularization in Two Layer Neural Networks

Traditional and Heavy-Tailed Self Regularization in Neural Network Models

The Prevalence of Neural Collapse in Neural Multivariate Regression

Stochastic collapse: how gradient noise attracts SGD dynamics towards simpler subnetworks*