Abstract:State-of-the-art text classification models are dominated by deep neural networks, but they still struggle to the issue of poor generalization ability when using cross entropy loss for training. One of the reasons is the training samples are usually annotated with hard labels, ignoring the inter-label relationship among these labels. This results in the outputted distributions to violate the label-correlation. Although the widely used contrastive learning is able to learn highly-expressive feature representations that generalize well across the training and test sets, contrastive learning in the embedding space is difficult to model the label correlation and may still output unreasonable label distributions. In this paper, we suggest con-trastive learning using label distributions and present a novel label-level contrastive learning (LLCL) para-digm, which can constrain the unreasonable label distributions from model outputs. We hypothesize that the label distributions of the instances in the same class are more similar than those from other classes. We introduce two label-level contrastive learning losses, namely supervised contrastive learning and self-supervised contrastive learning. After adding our proposed losses to the cross-entropy loss as regu-larizer for the training text classification model, our model obtains the average improvement of 0.74% over the strong RoBERTa-Large baseline on ten datasets. Particularly, our contrastive learning in the label space can effectively capture label correlations better than that in the embedding space, achieving an improvement of 6.2% in the top-2 accuracy on the SST-5 dataset. We also demonstrate that our proposed method is more effective, especially in the text classification tasks with a large label space or limited labeled data. Last but not least, our model does not rely on any kind of specialized architectures, data aug-mentation methods, or additional unsupervised data.CO 2022 Elsevier B.V. All rights reserved.

Class-aware and Augmentation-free Contrastive Learning from Label Proportion

Learning under Label Proportions for Text Classification

Two-stage Training for Learning from Label Proportions

Learning from Label Proportions with Consistency Regularization

Learning from Label Proportions by Learning with Label Noise

Forming Auxiliary High-confident Instance-level Loss to Promote Learning from Label Proportions

OT-LLP: Optimal Transport for Learning from Label Proportions

MixBag: Bag-Level Data Augmentation for Learning from Label Proportions

LLP-Bench: A Large Scale Tabular Benchmark for Learning from Label Proportions

Contrastive Label Enhancement

Theoretical Proportion Label Perturbation for Learning from Label Proportions in Large Bags

Learning from Label Proportions: Bootstrapping Supervised Learners via Belief Propagation

Label Anchored Contrastive Learning for Language Understanding

Learning from Label Proportions and Covariate-shifted Instances

Tabular Data Contrastive Learning via Class-Conditioned and Feature-Correlation Based Augmentation

Imbalanced Augmented Class Learning with Unlabeled Data by Label Confidence Propagation

Contrastive Learning from Label Distribution: A Case Study on Text Classification

Learning from Label Proportions with Instance-wise Consistency

Dual Contrastive Label Enhancement

Learning with Label Proportions by Incorporating Unmarked Data.

Local Contrastive Feature learning for Tabular Data