Abstract:Complementary-label learning (CLL) is a weakly-supervised learning paradigm that aims to train a multi-class classifier using only complementary labels, which indicate classes to which an instance does not belong. Despite numerous algorithmic proposals for CLL, their practical applicability remains unverified for two reasons. Firstly, these algorithms often rely on assumptions about the generation of complementary labels, and it is not clear how far the assumptions are from reality. Secondly, their evaluation has been limited to synthetic datasets. To gain insights into the real-world performance of CLL algorithms, we developed a protocol to collect complementary labels from human annotators. Our efforts resulted in the creation of four datasets: CLCIFAR10, CLCIFAR20, CLMicroImageNet10, and CLMicroImageNet20, derived from well-known classification datasets CIFAR10, CIFAR100, and TinyImageNet200. These datasets represent the very first real-world CLL datasets. Through extensive benchmark experiments, we discovered a notable decrease in performance when transitioning from synthetic datasets to real-world datasets. We investigated the key factors contributing to the decrease with a thorough dataset-level ablation study. Our analyses highlight annotation noise as the most influential factor in the real-world datasets. In addition, we discover that the biased-nature of human-annotated complementary labels and the difficulty to validate with only complementary labels are two outstanding barriers to practical CLL. These findings suggest that the community focus more research efforts on developing CLL algorithms and validation schemes that are robust to noisy and biased complementary-label distributions.

Complementary label learning based on knowledge distillation

Uncertainty-aware Complementary Label Queries for Active Learning

Complementary Labels Learning with Augmented Classes

Enhancing Label Sharing Efficiency in Complementary-Label Learning with Label Augmentation

Self-adaptive label discovery and multi-view fusion for complementary label learning

CLImage: Human-Annotated Datasets for Complementary-Label Learning

Self-supervised Knowledge Distillation for Complementary Label Learning

ComCo: Complementary supervised contrastive learning for complementary label learning

Learning from Complementary Labels Via Partial-Output Consistency Regularization.

Class-Imbalanced Complementary-Label Learning via Weighted Loss

Learning From a Complementary-Label Source Domain: Theory and Algorithms

libcll: an Extendable Python Toolkit for Complementary-Label Learning

Complementary Classifier Induced Partial Label Learning

Discriminative Complementary-Label Learning with Weighted Loss.

Complementary to Multiple Labels: A Correlation-Aware Correction Approach

Learning with Multiple Complementary Labels

Online Knowledge Distillation Via Mutual Contrastive Learning for Visual Recognition

Dual-Regularization Complementary Learning for Image Classification.

GAN-CL: Generative Adversarial Networks for Learning from Complementary Labels

Generative-Discriminative Complementary Learning

Dual-Regularization Complementary Learning for Image Classification