Abstract:This paper explores the connection between two recently identified phenomena in deep learning: plasticity loss and neural collapse. We analyze their correlation in different scenarios, revealing a significant association during the initial training phase on the first task. Additionally, we introduce a regularization approach to mitigate neural collapse, demonstrating its effectiveness in alleviating plasticity loss in this specific setting.
What problem does this paper attempt to address?
The paper attempts to address the relationship between Plasticity Loss (PL) and Neural Collapse (NC). Specifically, the authors aim to explore the following points:
1. **The association between plasticity loss and neural collapse**: The authors hope to reveal the correlation between these two phenomena by analyzing experimental results in different scenarios. Particularly, whether these two phenomena are significantly related in the initial training phase.
2. **The impact of mitigating neural collapse on neural plasticity**: The authors propose a regularization method to alleviate neural collapse and evaluate its effectiveness in mitigating plasticity loss under specific settings.
### Background
- **Neural plasticity**: A concept originating from neuroscience, referring to the nervous system's ability to adapt to new experiences. In artificial neural networks, neural plasticity refers to the network's ability to adapt when faced with new tasks.
- **Plasticity loss**: Refers to the phenomenon where neural networks perform poorly on continuous learning or non-stationary targets, especially when dealing with new tasks.
- **Neural collapse**: A recently observed phenomenon characterized by the collapse of the last layer features towards class centers, which is particularly evident in modern training paradigms.
### Research Motivation
- **Challenges in continuous learning**: In continuous learning, neural networks need to constantly learn new tasks, and retraining for each new task is usually infeasible. Therefore, understanding and mitigating plasticity loss becomes particularly important.
- **Impact of neural collapse**: The authors hypothesize that neural collapse may affect the plasticity of neural networks, as the collapse of features towards class centers can lead to information loss, thereby affecting the model's adaptability.
### Experimental Design
- **Datasets and models**: The authors conducted experiments in two different continuous learning settings, namely Permuted MNIST and Warm starting.
- **Metrics**: The primary metrics used were the four neural collapse metrics proposed by Papyan et al., particularly NC1, to measure the intra-class variability of the last layer features.
### Main Findings
1. **Permuted MNIST experiment**:
- **Neural collapse in continuous learning**: As the task index increases, training accuracy decreases, NC1 metric increases, and the two are highly correlated (Pearson correlation coefficient of -0.94).
- **Impact of initial task training time**: The training time of the initial task is significantly related to plasticity loss, but after exceeding a certain threshold, neural collapse no longer worsens, while plasticity loss continues to increase.
2. **Warm starting experiment**:
- **Strong correlation in early training**: In the early stages of training, there is a strong positive correlation between neural collapse and plasticity loss.
- **Effect of neural collapse regularization**: Adding an NC1 regularization term during the warm-up phase can significantly improve test accuracy and mitigate plasticity loss.
### Conclusion
- **Relationship between plasticity loss and neural collapse**: In continuous learning settings, once plasticity loss occurs, neural collapse cannot happen. In cases where the first task can be overfitted, there is a significant positive correlation between neural collapse and plasticity loss, but this correlation weakens as the training time for the first task increases.
- **Experimental validation**: The relationship between neural collapse and plasticity loss was validated through experiments, and the effectiveness of mitigating plasticity loss through regularization methods was proposed.
### Future Research Directions
- **Variable influences**: Factors such as network size, optimization strategies, and the similarity of subsequent tasks can all affect the relationship between plasticity loss and neural collapse. Future research needs to further explore the impact of these variables.
- **Causal relationship**: Although experimental results suggest a possible causal relationship between neural collapse and plasticity loss, more research is needed to verify this.