A Unified Contrastive Loss for Self-Training

Aurelien Gauffre,Julien Horvat,Massih-Reza Amini

DOI: https://doi.org/10.1007/978-3-031-70371-3_1

2024-09-11

Abstract:Self-training methods have proven to be effective in exploiting abundant unlabeled data in semi-supervised learning, particularly when labeled data is scarce. While many of these approaches rely on a cross-entropy loss function (CE), recent advances have shown that the supervised contrastive loss function (SupCon) can be more effective. Additionally, unsupervised contrastive learning approaches have also been shown to capture high quality data representations in the unsupervised setting. To benefit from these advantages in a semi-supervised setting, we propose a general framework to enhance self-training methods, which replaces all instances of CE losses with a unique contrastive loss. By using class prototypes, which are a set of class-wise trainable parameters, we recover the probability distributions of the CE setting and show a theoretical equivalence with it. Our framework, when applied to popular self-training methods, results in significant performance improvements across three different datasets with a limited number of labeled data. Additionally, we demonstrate further improvements in convergence speed, transfer ability, and hyperparameter stability. The code is available at \url{<a class="link-external link-https" href="https://github.com/AurelienGauffre/semisupcon/" rel="external noopener nofollow">this https URL</a>}.

Machine Learning,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the problem of improving the effectiveness of self-training methods in semi-supervised learning by utilizing a contrastive loss function. Specifically, the paper proposes a unified Semi-Supervised Contrastive (SSC) framework, aiming to enhance model performance in scenarios with scarce data labels by combining the advantages of supervised and unsupervised contrastive learning.

The main contributions include:
1. Proposing a new semi-supervised learning framework based on a unified contrastive loss (SSC), which can simultaneously handle labeled data, pseudo-labeled data, and data where the model lacks confidence.
2. Introducing the concept of class prototypes and establishing a theoretical connection between traditional cross-entropy loss and supervised contrastive learning.
3. Applying this loss to existing self-training methods (such as FixMatch) and achieving significant performance improvements on three different datasets, demonstrating faster convergence, better transferability, and more stable hyperparameter performance.

Through these contributions, the paper demonstrates that using a unified contrastive loss in semi-supervised learning scenarios can effectively leverage unlabeled data, thereby improving the overall performance of the model when labeled data is limited.