Instance-dependent Label-noise Learning under a Structural Causal Model

Yu Yao,Tongliang Liu,Mingming Gong,Bo Han,Gang Niu,Kun Zhang

DOI: https://doi.org/10.48550/arXiv.2109.02986

2022-06-03

Abstract:Label noise will degenerate the performance of deep learning algorithms because deep neural networks easily overfit label errors. Let X and Y denote the instance and clean label, respectively. When Y is a cause of X, according to which many datasets have been constructed, e.g., SVHN and CIFAR, the distributions of P(X) and P(Y|X) are entangled. This means that the unsupervised instances are helpful to learn the classifier and thus reduce the side effect of label noise. However, it remains elusive on how to exploit the causal information to handle the label noise problem. In this paper, by leveraging a structural causal model, we propose a novel generative approach for instance-dependent label-noise learning. In particular, we show that properly modeling the instances will contribute to the identifiability of the label noise transition matrix and thus lead to a better classifier. Empirically, our method outperforms all state-of-the-art methods on both synthetic and real-world label-noise datasets.

Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **How to use causal models to improve the performance and robustness of deep - learning algorithms in the case of instance - dependent label noise**.

Specifically, label noise can degrade the performance of deep - learning algorithms because deep neural networks are prone to over - fitting wrong labels. When the clean label \(Y\) is the cause of instance \(X\), the distributions \(P(X)\) and \(P(Y|X)\) of many datasets (such as SVHN and CIFAR) are entangled. This means that unsupervised instances are helpful for learning classifiers and reducing the side effects of label noise. However, how to use causal information to deal with the label noise problem remains an unsolved mystery.

To solve this problem, the author proposes a new generation method based on the Structural Causal Model (SCM) for learning instance - dependent label noise. By modeling instances, the label noise transition matrix \(P(\tilde{Y}|Y,X)\) can be identified, thereby improving the performance of the classifier. Experimental results show that this method outperforms the existing state - of - the - art methods on both synthetic and real - world label - noise datasets.

### Key points summary:

1. **Problem background**:
- Label noise can lead to a decline in the performance of deep - learning models.
- Existing methods improve models by modeling the label noise transition matrix \(P(\tilde{Y}|Y,X)\), but these methods usually need to assume that the label noise is instance - independent, which is difficult to satisfy in practice.

2. **Proposed method**:
- Use the Structural Causal Model (SCM) to capture the causal relationships in the data generation process.
- Constrain instances by modeling \(P(X|Y,Z)\) to help identify the label noise transition matrix \(P(\tilde{Y}|Y,X)\).
- Propose a causal generative model based on the Variational Auto - Encoder (VAE) framework, which can effectively infer the latent variables \(Z\) and \(Y\) from noisy data.

3. **Experimental verification**:
- Experiments were carried out on multiple synthetic and real - world datasets, including FashionMNIST, SVHN, CIFAR - 10 and CIFAR - 100.
- The experimental results show that the proposed method performs particularly well in the case of a high noise rate and significantly outperforms existing methods.

Through this method, the author not only improves the robustness of the model under label noise, but also provides a new perspective to understand and deal with the label noise problem.