Abstract:Machine learning classifiers have been demonstrated, both empirically and theoretically, to be robust to label noise under certain conditions -- notably the typical assumption is that label noise is independent of the features given the class label. We provide a theoretical framework that generalizes beyond this typical assumption by modeling label noise as a distribution over feature space. We show that both the scale and the shape of the noise distribution influence the posterior likelihood; and the shape of the noise distribution has a stronger impact on classification performance if the noise is concentrated in feature space where the decision boundary can be moved. For the special case of uniform label noise (independent of features and the class label), we show that the Bayes optimal classifier for $c$ classes is robust to label noise until the ratio of noisy samples goes above $\frac{c-1}{c}$ (e.g. 90% for 10 classes), which we call the tipping point. However, for the special case of class-dependent label noise (independent of features given the class label), the tipping point can be as low as 50%. Most importantly, we show that when the noise distribution targets decision boundaries (label noise is directly dependent on feature space), classification robustness can drop off even at a small scale of noise. Even when evaluating recent label-noise mitigation methods we see reduced accuracy when label noise is dependent on features. These findings explain why machine learning often handles label noise well if the noise distribution is uniform in feature-space; yet it also points to the difficulty of overcoming label noise when it is concentrated in a region of feature space where a decision boundary can move.

Non-Convex Boosting Overcomes Random Label Noise

Boosting in the presence of label noise

A Label Noise Robust Stacked Auto-Encoder Algorithm for Inaccurate Supervised Classification Problems

A Study on the Noise Label Influence in Boosting Algorithms: AdaBoost, GBM and XGBoost

Analyze the Robustness of Classifiers under Label Noise

A more robust boosting algorithm

Robustness and Reliability When Training With Noisy Labels

Reliable Label Correction is a Good Booster When Learning with Extremely Noisy Labels.

Boosting in the presence of outliers: adaptive classification with non-convex loss functions

Making Risk Minimization Tolerant to Label Noise

Open-set Label Noise Can Improve Robustness Against Inherent Label Noise

Convex and Non-convex Approaches for Statistical Inference with Class-Conditional Noisy Labels

Robustness to Label Noise Depends on the Shape of the Noise Distribution in Feature Space

Learning With Non-Uniform Label Noise: A Cluster-Dependent Weakly Supervised Approach.

Cross-Validation Is All You Need: A Statistical Approach To Label Noise Estimation

Soft Margins for AdaBoost

Contrastive Learning Improves Model Robustness Under Label Noise

Label-Noise Robust Logistic Regression and Its Applications

Loss factorization, weakly supervised learning and label noise robustness

Research of AdaBoost robustness based on Learning Automata

An Ensemble Noise-Robust K-fold Cross-Validation Selection Method for Noisy Labels