Abstract:Complementary-label learning (CLL) is a weakly-supervised learning paradigm that aims to train a multi-class classifier using only complementary labels, which indicate classes to which an instance does not belong. Despite numerous algorithmic proposals for CLL, their practical applicability remains unverified for two reasons. Firstly, these algorithms often rely on assumptions about the generation of complementary labels, and it is not clear how far the assumptions are from reality. Secondly, their evaluation has been limited to synthetic datasets. To gain insights into the real-world performance of CLL algorithms, we developed a protocol to collect complementary labels from human annotators. Our efforts resulted in the creation of four datasets: CLCIFAR10, CLCIFAR20, CLMicroImageNet10, and CLMicroImageNet20, derived from well-known classification datasets CIFAR10, CIFAR100, and TinyImageNet200. These datasets represent the very first real-world CLL datasets. Through extensive benchmark experiments, we discovered a notable decrease in performance when transitioning from synthetic datasets to real-world datasets. We investigated the key factors contributing to the decrease with a thorough dataset-level ablation study. Our analyses highlight annotation noise as the most influential factor in the real-world datasets. In addition, we discover that the biased-nature of human-annotated complementary labels and the difficulty to validate with only complementary labels are two outstanding barriers to practical CLL. These findings suggest that the community focus more research efforts on developing CLL algorithms and validation schemes that are robust to noisy and biased complementary-label distributions.

Learning from Complementary Labels Via Partial-Output Consistency Regularization.

Uncertainty-aware Complementary Label Queries for Active Learning

Discriminative Complementary-Label Learning with Weighted Loss.

Complementary Labels Learning with Augmented Classes

Complementary to Multiple Labels: A Correlation-Aware Correction Approach

Class-Imbalanced Complementary-Label Learning via Weighted Loss

Learning with Multiple Complementary Labels

Complementary label learning based on knowledge distillation

Complementary Classifier Induced Partial Label Learning

Enhancing Label Sharing Efficiency in Complementary-Label Learning with Label Augmentation

ComCo: Complementary supervised contrastive learning for complementary label learning

Biased Complementary-Label Learning Without True Labels

Self-adaptive label discovery and multi-view fusion for complementary label learning

Dual-Regularization Complementary Learning for Image Classification.

Learning with Complementary Labels Revisited: The Selected-Completely-at-Random Setting Is More Practical

Revisiting Consistency Regularization for Deep Partial Label Learning.

Dual-Regularization Complementary Learning for Image Classification

CLImage: Human-Annotated Datasets for Complementary-Label Learning

Unbiased Risk Estimator to Multi-Labeled Complementary Label Learning

Controller-Guided Partial Label Consistency Regularization with Unlabeled Data

Learning with Biased Complementary Labels