Abstract:Recently it has been observed that neural networks exhibit Neural Collapse (NC) during the final stage of training for the classification problem. We empirically show that multivariate regression, as employed in imitation learning and other applications, exhibits Neural Regression Collapse (NRC), a new form of neural collapse: (NRC1) The last-layer feature vectors collapse to the subspace spanned by the $n$ principal components of the feature vectors, where $n$ is the dimension of the targets (for univariate regression, $n=1$); (NRC2) The last-layer feature vectors also collapse to the subspace spanned by the last-layer weight vectors; (NRC3) The Gram matrix for the weight vectors converges to a specific functional form that depends on the covariance matrix of the targets. After empirically establishing the prevalence of (NRC1)-(NRC3) for a variety of datasets and network architectures, we provide an explanation of these phenomena by modeling the regression task in the context of the Unconstrained Feature Model (UFM), in which the last layer feature vectors are treated as free variables when minimizing the loss function. We show that when the regularization parameters in the UFM model are strictly positive, then (NRC1)-(NRC3) also emerge as solutions in the UFM optimization problem. We also show that if the regularization parameters are equal to zero, then there is no collapse. To our knowledge, this is the first empirical and theoretical study of neural collapse in the context of regression. This extension is significant not only because it broadens the applicability of neural collapse to a new category of problems but also because it suggests that the phenomena of neural collapse could be a universal behavior in deep learning.

Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networks

Preventing Representational Rank Collapse in MPNNs by Splitting the Computational Graph

A Neural Collapse Perspective on Feature Evolution in Graph Neural Networks

Simplifying the Theory on Over-Smoothing

Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?

The Persistence of Neural Collapse Despite Low-Rank Bias: An Analytic Perspective Through Unconstrained Features

A Survey on Oversmoothing in Graph Neural Networks

Beyond smoothness: A general optimization framework for graph neural networks with negative Laplacian regularization

Feature Correlation Aggregation: on the Path to Better Graph Neural Networks

Rank Diminishing in Deep Neural Networks

Graph Neural Networks Exponentially Lose Expressive Power for Node Classification

Graph Neural Networks Do Not Always Oversmooth

The Prevalence of Neural Collapse in Neural Multivariate Regression

Neural Rank Collapse: Weight Decay and Small Within-Class Variability Yield Low-Rank Bias

Distilling Influences to Mitigate Prediction Churn in Graph Neural Networks

Limitations of Neural Collapse for Understanding Generalization in Deep Learning

Breaking the Expressive Bottlenecks of Graph Neural Networks

Low-Rank Learning by Design: the Role of Network Architecture and Activation Linearity in Gradient Rank Collapse

Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networks

Oversquashing in GNNs through the lens of information contraction and graph expansion

Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNs