Generalized Supervised Contrastive Learning

Jaewon Kim,Hyukjong Lee,Jooyoung Chang,Sang Min Park

DOI: https://doi.org/10.48550/arXiv.2206.00384

2023-05-22

Abstract:With the recent promising results of contrastive learning in the self-supervised learning paradigm, supervised contrastive learning has successfully extended these contrastive approaches to supervised contexts, outperforming cross-entropy on various datasets. However, supervised contrastive learning inherently employs label information in a binary form--either positive or negative--using a one-hot target vector. This structure struggles to adapt to methods that exploit label information as a probability distribution, such as CutMix and knowledge distillation. In this paper, we introduce a generalized supervised contrastive loss, which measures cross-entropy between label similarity and latent similarity. This concept enhances the capabilities of supervised contrastive loss by fully utilizing the label distribution and enabling the adaptation of various existing techniques for training modern neural networks. Leveraging this generalized supervised contrastive loss, we construct a tailored framework: the Generalized Supervised Contrastive Learning (GenSCL). Compared to existing contrastive learning frameworks, GenSCL incorporates additional enhancements, including advanced image-based regularization techniques and an arbitrary teacher classifier. When applied to ResNet50 with the Momentum Contrast technique, GenSCL achieves a top-1 accuracy of 77.3% on ImageNet, a 4.1% relative improvement over traditional supervised contrastive learning. Moreover, our method establishes new state-of-the-art accuracies of 98.2% and 87.0% on CIFAR10 and CIFAR100 respectively when applied to ResNet50, marking the highest reported figures for this architecture.

Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to make full use of label distribution information in supervised contrastive learning to adapt to modern regularization methods such as CutMix and knowledge distillation. Specifically, traditional supervised contrastive learning methods use a binary form (i.e., positive samples or negative samples) when using label information, which makes it difficult to combine with methods that utilize label probability distributions. The paper proposes a Generalized Supervised Contrastive Loss (GenSCL), which overcomes this limitation by measuring the cross - entropy between label similarity and latent feature similarity. This method can not only make more comprehensive use of label information, but also enable the supervised contrastive learning framework to better adapt to various existing image augmentation techniques and arbitrary teacher classifiers, thereby improving the learning efficiency and accuracy of the model.

The main contributions of the paper include:
1. Proposing a new extension of the supervised contrastive loss function, which can fully utilize the probability distribution form of labels and defines the consistency between the latent feature space and the label space.
2. Introducing a customized contrastive learning framework (GenSCL) based on generalized supervised contrastive learning, which can seamlessly adapt to mixing techniques (such as CutMix) and knowledge distillation.
3. Analyzing and empirically demonstrating the effects of mixing techniques in contrastive training. These techniques generate hard positive samples and prevent the gradient from disappearing during the training process.
4. Empirically showing that the latent features learned through GenSCL are more decoupled.

These contributions aim to improve the performance of supervised contrastive learning and enable it to better combine with modern deep - learning techniques, thus achieving significant performance improvements on multiple benchmark datasets.