Abstract:We consider a crowd-sourcing problem where in the process of labeling massive datasets, multiple labelers with unknown annotation quality must be selected to perform the labeling task for each incoming data sample or task, with the results aggregated using for example simple or weighted majority voting rule. In this paper we approach this labeler selection problem in an online learning framework, whereby the quality of the labeling outcome by a specific set of labelers is estimated so that the learning algorithm over time learns to use the most effective combinations of labelers. This type of online learning in some sense falls under the family of multi-armed bandit (MAB) problems, but with a distinct feature not commonly seen: since the data is unlabeled to begin with and the labelers' quality is unknown, their labeling outcome (or reward in the MAB context) cannot be directly verified; it can only be estimated against the crowd and known probabilistically. We design an efficient online algorithm LS_OL using a simple majority voting rule that can differentiate high- and low-quality labelers over time, and is shown to have a regret (w.r.t. always using the optimal set of labelers) of O(log 2 T) uniformly in time under mild assumptions on the collective quality of the crowd, thus regret free in the average sense. We discuss performance improvement by using a more sophisticated majority voting rule, and show how to detect and filter out "bad" (dishonest, malicious or very incompetent) labelers to further enhance the quality of crowd-sourcing. Extension to the case when a labeler's quality is task-type dependent is also discussed using techniques from the literature on continuous arms. We present numerical results using both simulation and a real dataset on a set of images labeled by Amazon Mechanic Turks (AMT).

Improving the Quality of Crowdsourcing Labels by Combination of Golden Data and Incentive

Learning from Crowds under Experts' Supervision

A Formalized Framework for Incorporating Expert Labels in Crowdsourcing Environment

Task Assignment with Guaranteed Quality for Crowdsourcing Platforms.

Crowdsourcing Label Quality: A Theoretical Analysis

Label Consistency-Based Ground Truth Inference for Crowdsourcing

Recovering Missing Labels of Crowdsourcing Workers.

Hierarchical Crowdsourcing for Data Labeling with Heterogeneous Crowd.

An Online Learning Approach to Improving the Quality of Crowd-Sourcing

Optimizing the Wisdom of the Crowd: Inference, Learning, and Teaching

Human-centred Design on Crowdsourcing Annotation Towards Improving Active Learning Model Performance

Quality-Aware Incentive Mechanisms Under Social Influences in Data Crowdsourcing

Inference Aided Reinforcement Learning for Incentive Mechanism Design in Crowdsourcing

Collusion Detection and Ground Truth Inference in Crowdsourcing for Labeling Tasks.

Crowdsourcing Truth Inference Based on Label Confidence Clustering

Labelling Training Samples Using Crowdsourcing Annotation for Recommendation

Reverse-auction-based Crowdsourced Labeling for Active Learning.

LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systems

Crowdsourced POI Labelling: Location-aware Result Inference and Task Assignment.

Modeling for Noisy Labels of Crowd Workers.

Obtaining High-Quality Label by Distinguishing Between Easy and Hard Items in Crowdsourcing