Abstract:We consider a crowd-sourcing problem where in the process of labeling massive datasets, multiple labelers with unknown annotation quality must be selected to perform the labeling task for each incoming data sample or task, with the results aggregated using for example simple or weighted majority voting rule. In this paper we approach this labeler selection problem in an online learning framework, whereby the quality of the labeling outcome by a specific set of labelers is estimated so that the learning algorithm over time learns to use the most effective combinations of labelers. This type of online learning in some sense falls under the family of multi-armed bandit (MAB) problems, but with a distinct feature not commonly seen: since the data is unlabeled to begin with and the labelers' quality is unknown, their labeling outcome (or reward in the MAB context) cannot be directly verified; it can only be estimated against the crowd and known probabilistically. We design an efficient online algorithm LS_OL using a simple majority voting rule that can differentiate high- and low-quality labelers over time, and is shown to have a regret (w.r.t. always using the optimal set of labelers) of O(log 2 T) uniformly in time under mild assumptions on the collective quality of the crowd, thus regret free in the average sense. We discuss performance improvement by using a more sophisticated majority voting rule, and show how to detect and filter out "bad" (dishonest, malicious or very incompetent) labelers to further enhance the quality of crowd-sourcing. Extension to the case when a labeler's quality is task-type dependent is also discussed using techniques from the literature on continuous arms. We present numerical results using both simulation and a real dataset on a set of images labeled by Amazon Mechanic Turks (AMT).

Joint optimization of scoring and thresholding models for online multi-label classification

Adaptive label thresholding methods for online multi-label classification.

Online Passive-Aggressive Multilabel Classification Algorithms

Online Multi-Label Classification under Noisy and Changing Label Distribution

Online active classification via margin-based and feature-based label queries

Two-dimensional multilabel active learning with an efficient online adaptation model for image classification.

Parallel Algorithm for Optimal Threshold Labeling of Ordinal Regression Methods

Efficient Methods for Non-stationary Online Learning

Optimizing Class Distribution in Memory for Multi-Label Online Continual Learning

Adaptive Cost-Sensitive Online Classification

Learning Label Correlations for Multi-Label Online Passive Aggressive Classification Algorithm

Joint Ranking SVM and Binary Relevance with Robust Low-Rank Learning for Multi-Label Classification

A General Online Algorithm for Optimizing Complex Performance Metrics

An Online Learning Approach to Improving the Quality of Crowd-Sourcing

Joint coordinate attention mechanism and instance normalization for COVID online comments text classification

Online Multi-threshold Learning with Imbalanced Data Stream.

Joint Optimization Framework for Learning with Noisy Labels

Dual-Decoupling Learning and Metric-Adaptive Thresholding for Semi-Supervised Multi-Label Learning

Joint Multi-Label Classification and Label Correlations with Missing Labels and Feature Selection

Online Feature Updates Improve Online (Generalized) Label Shift Adaptation