Abstract:Existing research on learning with noisy labels mainly focuses on synthetic label noise. Synthetic noise, though has clean structures which greatly enabled statistical analyses, often fails to model real-world noise patterns. The recent literature has observed several efforts to offer real-world noisy datasets, yet the existing efforts suffer from two caveats: (1) The lack of ground-truth verification makes it hard to theoretically study the property and treatment of real-world label noise; (2) These efforts are often of large scales, which may result in unfair comparisons of robust methods within reasonable and accessible computation power. To better understand real-world label noise, it is crucial to build controllable and moderate-sized real-world noisy datasets with both ground-truth and noisy labels. This work presents two new benchmark datasets CIFAR-10N, CIFAR-100N, equipping the training datasets of CIFAR-10, CIFAR-100 with human-annotated real-world noisy labels we collected from Amazon Mechanical Turk. We quantitatively and qualitatively show that real-world noisy labels follow an instance-dependent pattern rather than the classically assumed and adopted ones (e.g., class-dependent label noise). We then initiate an effort to benchmarking a subset of the existing solutions using CIFAR-10N and CIFAR-100N. We further proceed to study the memorization of correct and wrong predictions, which further illustrates the difference between human noise and class-dependent synthetic noise. We show indeed the real-world noise patterns impose new and outstanding challenges as compared to synthetic label noise. These observations require us to rethink the treatment of noisy labels, and we hope the availability of these two datasets would facilitate the development and evaluation of future learning with noisy label solutions. Datasets and leaderboards are available at <a class="link-external link-http" href="http://noisylabels.com" rel="external noopener nofollow">this http URL</a>.

Performance of Classifiers on Noisy-Labeled Training Data: An Empirical Study on Handwritten Digit Classification Task

FGCM: Noisy Label Learning via Fine-Grained Confidence Modeling

Noisy Label Processing for Classification: A Survey

Comparative Study on Handwritten Digit Recognition Classifier Using CNN and Machine Learning Algorithms

Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey

A Novel Handwritten Digit Classification System Based on Convolutional Neural Network Approach

Handwritten digit recognition based on classical machine learning methods

Learning Image Labels On-the-fly for Training Robust Classification Models

Deep Learning Classification With Noisy Labels

Learning with Noisy Labels Via Self-supervised Adversarial Noisy Masking

Learning Sound Event Classifiers from Web Audio with Noisy Labels

Task-Adaptive Pre-Training for Boosting Learning With Noisy Labels: A Study on Text Classification for African Languages

Classification Performance Analysis of Decision Tree-Based Algorithms with Noisy Class Variable

Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations

Multiclass Learning from Noisy Labels for Non-decomposable Performance Measures

Model-agnostic Approaches to Handling Noisy Labels When Training Sound Event Classifiers

Decoding class dynamics in learning with noisy labels

Recognition of Handwritten Digit using Convolutional Neural Network in Python with Tensorflow and Comparison of Performance for Various Hidden Layers

Label noise and self-learning label correction in cardiac abnormalities classification.

Pruning Distorted Images in MNIST Handwritten Digits

Handwritten Digit Recognition using Machine and Deep Learning Algorithms