Abstract:We consider a crowd-sourcing problem where in the process of labeling massive datasets, multiple labelers with unknown annotation quality must be selected to perform the labeling task for each incoming data sample or task, with the results aggregated using for example simple or weighted majority voting rule. In this paper we approach this labeler selection problem in an online learning framework, whereby the quality of the labeling outcome by a specific set of labelers is estimated so that the learning algorithm over time learns to use the most effective combinations of labelers. This type of online learning in some sense falls under the family of multi-armed bandit (MAB) problems, but with a distinct feature not commonly seen: since the data is unlabeled to begin with and the labelers' quality is unknown, their labeling outcome (or reward in the MAB context) cannot be directly verified; it can only be estimated against the crowd and known probabilistically. We design an efficient online algorithm LS_OL using a simple majority voting rule that can differentiate high- and low-quality labelers over time, and is shown to have a regret (w.r.t. always using the optimal set of labelers) of O(log 2 T) uniformly in time under mild assumptions on the collective quality of the crowd, thus regret free in the average sense. We discuss performance improvement by using a more sophisticated majority voting rule, and show how to detect and filter out "bad" (dishonest, malicious or very incompetent) labelers to further enhance the quality of crowd-sourcing. Extension to the case when a labeler's quality is task-type dependent is also discussed using techniques from the literature on continuous arms. We present numerical results using both simulation and a real dataset on a set of images labeled by Amazon Mechanic Turks (AMT).

Cost-Effective Active Learning from Diverse Labelers.

Data : Labeler 1 : Labeler 2 : Labeler 3 : Figure

Cost-Accuracy Aware Adaptive Labeling for Active Learning

Uncertainty-aware Complementary Label Queries for Active Learning

Active Multi-label Learning with Optimal Label Subset Selection

Cost Effective Multi-label Active Learning Via Querying Subexamples

Human-centred Design on Crowdsourcing Annotation Towards Improving Active Learning Model Performance

Learning to Label with Active Learning and Reinforcement Learning.

Learning from Crowds under Experts' Supervision

Dual-Classifier Collaborative Method Based on Semi-Supervised Active Learning

Multi-label Active Learning: Query Type Matters

Sample Diversity Selection Strategy Based on Label Distribution Morphology for Active Label Distribution Learning

Active Learning from Weak and Strong Labelers

Active Learning Without Knowing Individual Instance Labels: A Pairwise Label Homogeneity Query Approach.

Active learning with confidence-based answers for crowdsourcing labeling tasks.

ActiveLab: Active Learning with Re-Labeling by Multiple Annotators

Active Query Driven By Uncertainty And Diversity For Incremental Multi-Label Learning

Asynchronous Active Learning with Distributed Label Querying

Robust and Discriminative Labeling for Multi-label Active Learning Based on Maximum Correntropy Criterion

Multi-label Active Learning Based on Submodular Functions

An Online Learning Approach to Improving the Quality of Crowd-Sourcing