Abstract:Accurately annotating large scale dataset is notoriously expensive both in time and in money. Although acquiring low-quality-annotated dataset can be much cheaper, it often badly damages the performance of trained models when using such dataset without particular treatment. Various methods have been proposed for learning with noisy labels. However, most methods only handle limited kinds of noise patterns, require auxiliary information or steps (e.g. , knowing or estimating the noise transition matrix), or lack theoretical justification. In this paper, we propose a novel information-theoretic loss function, $\mathcal{L}_{DMI}$, for training deep neural networks robust to label noise. The core of $\mathcal{L}_{DMI}$ is a generalized version of mutual information, termed Determinant based Mutual Information (DMI), which is not only information-monotone but also relatively invariant. \emph{To the best of our knowledge, $\mathcal{L}_{DMI}$ is the first loss function that is provably robust to instance-independent label noise, regardless of noise pattern, and it can be applied to any existing classification neural networks straightforwardly without any auxiliary information}. In addition to theoretical justification, we also empirically show that using $\mathcal{L}_{DMI}$ outperforms all other counterparts in the classification task on both image dataset and natural language dataset include Fashion-MNIST, CIFAR-10, Dogs vs. Cats, MR with a variety of synthesized noise patterns and noise amounts, as well as a real-world dataset Clothing1M. Codes are available at <a class="link-external link-https" href="https://github.com/Newbeeer/L_DMI" rel="external noopener nofollow">this https URL</a> .

Trimmed Robust Loss Function for Training Deep Neural Networks with Label Noise

Robust Loss Functions under Label Noise for Deep Neural Networks

FGCM: Noisy Label Learning via Fine-Grained Confidence Modeling

Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach

Noise is the Fatal Poison: A Noise-aware Network for Noisy Dataset Classification

Normalized Loss Functions for Deep Learning with Noisy Labels

Robustness and Reliability When Training With Noisy Labels

Learning with Noisy Labels Via Sparse Regularization

Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels

L_DMI: A Novel Information-theoretic Loss Function for Training Deep Nets Robust to Label Noise

Can Cross Entropy Loss Be Robust to Label Noise?

L_DMI: An Information-theoretic Noise-robust Loss Function

Noise-Robust Loss Functions: Enhancing Bounded Losses for Large-Scale Noisy Data Learning

Active Negative Loss: A Robust Framework for Learning with Noisy Labels

Asymmetric Loss Functions for Learning with Noisy Labels

Multi-category classification with label noise by robust binary loss

Symmetric Cross Entropy for Robust Learning With Noisy Labels

Learning with Noisy Labels Via Self-supervised Adversarial Noisy Masking