Abstract:Recently, deep learning models have been widely applied in program understanding tasks, and these models achieve state-of-the-art results on many benchmark datasets. A major challenge of deep learning for program understanding is that the effectiveness of these approaches depends on the quality of their datasets, and these datasets often contain noisy data samples. A typical kind of noise in program understanding datasets is label noise, which means that the target outputs for some inputs are incorrect. Researchers have proposed various approaches to alleviate the negative impact of noisy labels, and formed a new research topic: noisy label learning (NLL). In this paper, we conduct an empirical study on the effectiveness of noisy label learning on deep learning for program understanding datasets. We evaluate various NLL approaches and deep learning models on three tasks: program classification, vulnerability detection, and code summarization. From the evaluation results, we come to the following findings: 1) small trained-from-scratch models are prone to label noises in program understanding, while large pre-trained models are highly robust against them. 2) NLL approaches significantly improve the program classification accuracies for small models on noisy training sets, but they only slightly benefit large pre-trained models in classification accuracies. 3) NLL can effectively detect synthetic noises in program understanding, but struggle in detecting real-world noises. We believe our findings can provide insights on the abilities of NLL in program understanding, and shed light on future works in tackling noises in software engineering datasets. We have released our code at <a class="link-external link-https" href="https://github.com/jacobwwh/noise_SE" rel="external noopener nofollow">this https URL</a>.

Impact of Label Noise on Learning Complex Features

Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks

A Convergence Path to Deep Learning on Noisy Labels

Mitigating Instance-Dependent Label Noise: Integrating Self-Supervised Pretraining with Pseudo-Label Refinement

Dimensionality-Driven Learning with Noisy Labels.

Decoding class dynamics in learning with noisy labels

Limited Gradient Descent: Learning With Noisy Labels

Learning with Feature-Dependent Label Noise: A Progressive Approach

Learning with Noisy Foundation Models

Label Noise: Ignorance Is Bliss

Understanding Instance-Level Label Noise: Disparate Impacts and Treatments

An Empirical Study on Noisy Label Learning for Program Understanding

Which Strategies Matter for Noisy Label Classification? Insight into Loss and Uncertainty

On Emergence of Clean-Priority Learning in Early Stopped Neural Networks

Doubly Stochastic Models: Learning with Unbiased Label Noises and Inference Stability

Label Noise: Correcting the Forward-Correction

How benign is benign overfitting?

Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis

Model-agnostic Approaches to Handling Noisy Labels When Training Sound Event Classifiers

Adaptive Textual Label Noise Learning Based on Pre-trained Models

Learning Confident Classifiers in the Presence of Label Noise