Abstract:Complementary-label learning (CLL) is a weakly-supervised learning paradigm that aims to train a multi-class classifier using only complementary labels, which indicate classes to which an instance does not belong. Despite numerous algorithmic proposals for CLL, their practical applicability remains unverified for two reasons. Firstly, these algorithms often rely on assumptions about the generation of complementary labels, and it is not clear how far the assumptions are from reality. Secondly, their evaluation has been limited to synthetic datasets. To gain insights into the real-world performance of CLL algorithms, we developed a protocol to collect complementary labels from human annotators. Our efforts resulted in the creation of four datasets: CLCIFAR10, CLCIFAR20, CLMicroImageNet10, and CLMicroImageNet20, derived from well-known classification datasets CIFAR10, CIFAR100, and TinyImageNet200. These datasets represent the very first real-world CLL datasets. Through extensive benchmark experiments, we discovered a notable decrease in performance when transitioning from synthetic datasets to real-world datasets. We investigated the key factors contributing to the decrease with a thorough dataset-level ablation study. Our analyses highlight annotation noise as the most influential factor in the real-world datasets. In addition, we discover that the biased-nature of human-annotated complementary labels and the difficulty to validate with only complementary labels are two outstanding barriers to practical CLL. These findings suggest that the community focus more research efforts on developing CLL algorithms and validation schemes that are robust to noisy and biased complementary-label distributions.

Discriminative Complementary-Label Learning with Weighted Loss.

Uncertainty-aware Complementary Label Queries for Active Learning

Class-Imbalanced Complementary-Label Learning via Weighted Loss

Learning from Complementary Labels Via Partial-Output Consistency Regularization.

Complementary to Multiple Labels: A Correlation-Aware Correction Approach

Generative-Discriminative Complementary Learning

Complementary Labels Learning with Augmented Classes

ComCo: Complementary supervised contrastive learning for complementary label learning

Biased Complementary-Label Learning Without True Labels

Complementary Classifier Induced Partial Label Learning

Complementary label learning based on knowledge distillation

Unbiased Risk Estimator to Multi-Labeled Complementary Label Learning

Learning with Multiple Complementary Labels

Enhancing Label Sharing Efficiency in Complementary-Label Learning with Label Augmentation

Self-adaptive label discovery and multi-view fusion for complementary label learning

Multi-Complementary and Unlabeled Learning for Arbitrary Losses and Models

Simple and Effective Complementary Label Learning Based on Mean Square Error Loss

Multi-Complementary and Unlabeled Learning for Arbitrary Losses and Models

Extending Ordinary-Label Learning Losses to Complementary-Label Learning.

CLImage: Human-Annotated Datasets for Complementary-Label Learning

Learning with Biased Complementary Labels