Abstract:The presence of label noise in the training data has a profound impact on the generalization of deep neural networks (DNNs). In this study, we introduce and theoretically demonstrate a simple feature noise method, which directly adds noise to the features of training data, can enhance the generalization of DNNs under label noise. Specifically, we conduct theoretical analyses to reveal that label noise leads to weakened DNN generalization by loosening the PAC-Bayes generalization bound, and feature noise results in better DNN generalization by imposing an upper bound on the mutual information between the model weights and the features, which constrains the PAC-Bayes generalization bound. Furthermore, to ensure effective generalization of DNNs in the presence of label noise, we conduct application analyses to identify the optimal types and levels of feature noise to add for obtaining desirable label noise generalization. Finally, extensive experimental results on several popular datasets demonstrate the feature noise method can significantly enhance the label noise generalization of the state-of-the-art label noise method.

What problem does this paper attempt to address?

This paper attempts to address the problem of how to improve the generalization ability of deep neural networks (DNNs) in the presence of label noise in the training data. Specifically, the paper proposes a simple method to enhance the generalization performance of DNNs in a label noise environment by adding noise to the features of the training data. ### Background and Motivation - **Label Noise Problem**: Label noise refers to the presence of incorrectly labeled samples in the training dataset. In deep learning, due to the typically over-parameterized nature of deep neural networks, label noise significantly impacts the model's generalization ability. DNNs, with enough parameters and sufficient training, can even fit random label data. - **Existing Methods**: Existing methods for handling label noise can be mainly divided into two categories: - **Denoising Methods**: Aim to identify incorrectly labeled samples in the training set and mitigate their negative impact on DNNs. These methods rely on specific information to distinguish correctly labeled samples from incorrectly labeled ones. - **Robustness Methods**: Improve generalization ability by enhancing the overall robustness of the model without specifically focusing on noisy labels. These methods typically involve adding noise during training, such as data augmentation or algorithmic noise. ### Main Contributions of the Paper - **Proposing the Feature Noise Method**: The paper proposes a new method to improve the generalization ability of DNNs in a label noise environment by adding noise to the features of the training data. This method is simple, cost-effective, and applicable to different models. - **Theoretical Analysis**: The paper theoretically demonstrates the effectiveness of the feature noise method. Specifically, feature noise constrains the mutual information between model weights and features, thereby tightening the PAC-Bayes generalization bound and improving the model's generalization ability. - **Experimental Validation**: The paper conducts extensive experiments on multiple popular datasets, showing that the feature noise method significantly improves the generalization performance of DNNs in a label noise environment, even surpassing existing state-of-the-art methods. ### Theoretical Analysis - **Impact of Label Noise**: The paper first demonstrates that label noise relaxes the PAC-Bayes generalization bound, leading to a decline in the model's generalization ability. - **Role of Feature Noise**: The paper further proves that adding Gaussian noise to training features can tighten the PAC-Bayes generalization bound, thereby improving the generalization ability of DNNs in a label noise environment. ### Experimental Results - **Datasets**: The paper conducts experiments on multiple real-world datasets to validate the effectiveness and theoretical advantages of the feature noise method. - **Performance Comparison**: Experimental results show that the feature noise method achieves comparable or even better performance than existing state-of-the-art methods in a label noise environment. ### Conclusion Through theoretical analysis and experimental validation, the paper demonstrates that the feature noise method significantly improves the generalization ability of DNNs in a label noise environment. This method is not only simple and efficient but also provides a new perspective for addressing the label noise problem.

Feature Noise Boosts DNN Generalization under Label Noise