Abstract:Complementary-label learning (CLL) is a weakly-supervised learning paradigm that aims to train a multi-class classifier using only complementary labels, which indicate classes to which an instance does not belong. Despite numerous algorithmic proposals for CLL, their practical applicability remains unverified for two reasons. Firstly, these algorithms often rely on assumptions about the generation of complementary labels, and it is not clear how far the assumptions are from reality. Secondly, their evaluation has been limited to synthetic datasets. To gain insights into the real-world performance of CLL algorithms, we developed a protocol to collect complementary labels from human annotators. Our efforts resulted in the creation of four datasets: CLCIFAR10, CLCIFAR20, CLMicroImageNet10, and CLMicroImageNet20, derived from well-known classification datasets CIFAR10, CIFAR100, and TinyImageNet200. These datasets represent the very first real-world CLL datasets. Through extensive benchmark experiments, we discovered a notable decrease in performance when transitioning from synthetic datasets to real-world datasets. We investigated the key factors contributing to the decrease with a thorough dataset-level ablation study. Our analyses highlight annotation noise as the most influential factor in the real-world datasets. In addition, we discover that the biased-nature of human-annotated complementary labels and the difficulty to validate with only complementary labels are two outstanding barriers to practical CLL. These findings suggest that the community focus more research efforts on developing CLL algorithms and validation schemes that are robust to noisy and biased complementary-label distributions.

ComCo: Complementary supervised contrastive learning for complementary label learning

Uncertainty-aware Complementary Label Queries for Active Learning

Complementary label learning based on knowledge distillation

Learning from Complementary Labels Via Partial-Output Consistency Regularization.

Enhancing Label Sharing Efficiency in Complementary-Label Learning with Label Augmentation

Discriminative Complementary-Label Learning with Weighted Loss.

Complementary to Multiple Labels: A Correlation-Aware Correction Approach

Complementary Labels Learning with Augmented Classes

CLImage: Human-Annotated Datasets for Complementary-Label Learning

Boosting Semi-Supervised Learning with Contrastive Complementary Labeling

Complementary Classifier Induced Partial Label Learning

Self-adaptive label discovery and multi-view fusion for complementary label learning

Class-Imbalanced Complementary-Label Learning via Weighted Loss

GraphCoCo: Graph Complementary Contrastive Learning

Generative-Discriminative Complementary Learning

Learning with Multiple Complementary Labels

Label Anchored Contrastive Learning for Language Understanding

libcll: an Extendable Python Toolkit for Complementary-Label Learning

GAN-CL: Generative Adversarial Networks for Learning from Complementary Labels

CoMAL: Contrastive Active Learning for Multi-Label Text Classification

Biased Complementary-Label Learning Without True Labels