Abstract:Deep neural networks may easily memorize noisy labels present in real-world data, which degrades their ability to generalize. It is therefore important to track and evaluate the robustness of models against noisy label memorization. We propose a metric, called susceptibility, to gauge such memorization for neural networks. Susceptibility is simple and easy to compute during training. Moreover, it does not require access to ground-truth labels and it only uses unlabeled data. We empirically show the effectiveness of our metric in tracking memorization on various architectures and datasets and provide theoretical insights into the design of the susceptibility metric. Finally, we show through extensive experiments on datasets with synthetic and real-world label noise that one can utilize susceptibility and the overall training accuracy to distinguish models that maintain a low memorization on the training set and generalize well to unseen clean data.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the problem that deep neural networks are prone to memorize noisy labels during the training process, which will reduce the generalization ability of the model. Specifically: 1. **Problem background**: - When deep neural networks process real - world data, they often encounter data with noisy labels. These noisy labels may cause the model to over - fit, thus impairing its generalization performance. - Although existing methods can improve the robustness of the model to noisy labels, they usually need to access the true labels or a clean test set to evaluate the performance of the model. 2. **Research objectives**: - Propose a method without accessing the true labels to track and evaluate the model's memorization of noisy labels. - Design a simple and easy - to - calculate metric - **susceptibility** - to measure the degree of the model's memorization of noisy labels. - Verify the effectiveness of this metric through experiments and demonstrate its application on different architectures and datasets. 3. **Solutions**: - **Susceptibility metric**: This metric is based on the performance of the model on a held - out randomly - labeled set. Specifically, it measures the change in the objective function before and after a single optimization step. - **Theoretical analysis and empirical research**: Through theoretical analysis and a large number of experiments, it is proved that models with high test accuracy are more resistant to the memorization of noisy labels. In addition, a strong correlation between the susceptibility metric and the model's generalization performance is also shown. 4. **Practical applications**: - By using the susceptibility metric and the overall training accuracy, models that have less memorization of noisy labels and good generalization performance can be selected without accessing the true labels. - This method is applicable to a variety of datasets and model architectures, including datasets with synthetic noisy labels and real - world noisy labels. In summary, this paper proposes a novel evaluation method without true labels, which helps researchers better understand and control the behavior of deep learning models on noisy data, thereby improving the generalization ability of the model.

Leveraging Unlabeled Data to Track Memorization

Learn to Forget: Memorization Elimination for Neural Networks.

Detecting Memorization in Large Language Models

Memory Aware Synapses: Learning what (not) to forget

Uncovering Latent Memories: Assessing Data Leakage and Memorization Patterns in Frontier AI Models

Searching to Exploit Memorization Effect in Deep Learning with Noisy Labels.

Mitigating Memorization In Language Models

Measuring Catastrophic Forgetting in Neural Networks

Measuring Forgetting of Memorized Training Examples

Memorization in deep learning: A survey

Early-Learning Regularization Prevents Memorization of Noisy Labels

Confidence Adaptive Regularization for Deep Learning with Noisy Labels

Measuring memorization through probabilistic discoverable extraction

Exploring Memorization in Adversarial Training

MemHunter: Automated and Verifiable Memorization Detection at Dataset-scale in LLMs

Embedding Space Selection for Detecting Memorization and Fingerprinting in Generative Models

MILD: Modeling the Instance Learning Dynamics for Learning with Noisy Labels

Scalability of memorization-based machine unlearning

Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild

Multi-Objective Interpolation Training for Robustness to Label Noise

Learn to Forget: Machine Unlearning Via Neuron Masking