Abstract:Learning from noisy labels is a critical challenge in machine learning, with vast implications for numerous real-world scenarios. While supervised contrastive learning has recently emerged as a powerful tool for navigating label noise, many existing solutions remain heuristic, often devoid of a systematic theoretical foundation for crafting robust supervised contrastive losses. To address the gap, in this paper, we propose a unified theoretical framework for robust losses under the pairwise contrastive paradigm. In particular, we for the first time derive a general robust condition for arbitrary contrastive losses, which serves as a criterion to verify the theoretical robustness of a supervised contrastive loss against label noise. The theory indicates that the popular InfoNCE loss is in fact non-robust, and accordingly inspires us to develop a robust version of InfoNCE, termed Symmetric InfoNCE (SymNCE). Moreover, we highlight that our theory is an inclusive framework that provides explanations to prior robust techniques such as nearest-neighbor (NN) sample selection and robust contrastive loss. Validation experiments on benchmark datasets demonstrate the superiority of SymNCE against label noise.
What problem does this paper attempt to address?
The core problem that this paper attempts to solve is the robustness issue of supervised contrastive learning in the presence of label noise. Specifically, although supervised contrastive learning has shown potential in dealing with label noise, most of the existing methods lack a systematic theoretical basis to ensure their robustness. Therefore, this paper aims to establish a unified theoretical framework for designing and validating supervised contrastive loss functions that are robust to label noise.
### 1. Background and Motivation
Supervised learning has achieved remarkable success in many fields. However, in the real world, data labels are often noisy, that is, some data points are mislabeled. This label noise can seriously affect the performance of supervised models. Therefore, how to conduct effective learning in the presence of label noise has become an important research topic in machine learning.
In recent years, supervised contrastive learning has been introduced as a powerful tool to deal with the problem of label noise. However, most of the existing solutions rely on heuristic methods and lack systematic theoretical support. Therefore, there is an urgent need for a theoretical framework to guide the design of robust supervised contrastive loss functions.
### 2. Research Objectives
The main objectives of this paper are:
- **Establish a unified theoretical framework**: Derive the robustness conditions of any contrastive loss function in the presence of label noise, providing a theoretical basis for validating and designing robust supervised contrastive losses.
- **Propose a new robust loss function**: Based on the above - mentioned theoretical framework, develop a new robust loss function - Symmetric InfoNCE (SymNCE) - to deal with the impact of label noise.
- **Verify the effectiveness of the new method**: Verify the superior performance of SymNCE in the label - noise environment through experiments.
### 3. Main Contributions
The main contributions of this paper include:
- **For the first time, establish a unified theoretical framework for robust supervised contrastive losses**: Derive general robustness conditions applicable to any contrastive loss function and explain the principles of existing robust techniques such as nearest - neighbor sample selection and RINCE loss.
- **Propose Symmetric InfoNCE (SymNCE)**: By adding a carefully designed Reverse InfoNCE (RevNCE) to the widely - used InfoNCE loss function, make it robust to label noise.
- **Experimentally prove the effectiveness of SymNCE**: Experimental results on multiple benchmark datasets show that SymNCE significantly outperforms existing methods in the label - noise environment.
### 4. Method Overview
To achieve the above - mentioned objectives, this paper has carried out the following work:
#### 4.1 Derivation of Robustness Conditions
First, this paper derives the robustness conditions of any contrastive loss function in the presence of label noise. By analyzing the impact of label noise on the distribution of positive samples, the author finds that label noise can cause the true labels of positive samples to be different, thereby harming the effect of representation learning. Therefore, a robust contrastive loss should be able to eliminate this impact.
#### 4.2 Design of Symmetric InfoNCE
Based on the derived robustness conditions, this paper proposes Symmetric InfoNCE (SymNCE). Specifically, the author designs a Reverse InfoNCE (RevNCE) loss term and combines it with the original InfoNCE loss. This not only makes the total loss function satisfy the robustness conditions but also maintains the basic function of InfoNCE, that is, to align positive samples and push away negative samples.
#### 4.3 Experimental Verification
Finally, this paper verifies the effectiveness of SymNCE through a series of experiments. The experimental results show that under different noise rates, SymNCE always outperforms the existing supervised contrastive loss functions, especially in a high - noise environment.
In conclusion, this paper solves the robustness problem of supervised contrastive learning in the label - noise environment by establishing a theoretical framework and proposing a new robust loss function, providing important theoretical and technical support for future research.