Abstract:We consider a crowd-sourcing problem where in the process of labeling massive datasets, multiple labelers with unknown annotation quality must be selected to perform the labeling task for each incoming data sample or task, with the results aggregated using for example simple or weighted majority voting rule. In this paper we approach this labeler selection problem in an online learning framework, whereby the quality of the labeling outcome by a specific set of labelers is estimated so that the learning algorithm over time learns to use the most effective combinations of labelers. This type of online learning in some sense falls under the family of multi-armed bandit (MAB) problems, but with a distinct feature not commonly seen: since the data is unlabeled to begin with and the labelers' quality is unknown, their labeling outcome (or reward in the MAB context) cannot be directly verified; it can only be estimated against the crowd and known probabilistically. We design an efficient online algorithm LS_OL using a simple majority voting rule that can differentiate high- and low-quality labelers over time, and is shown to have a regret (w.r.t. always using the optimal set of labelers) of O(log 2 T) uniformly in time under mild assumptions on the collective quality of the crowd, thus regret free in the average sense. We discuss performance improvement by using a more sophisticated majority voting rule, and show how to detect and filter out "bad" (dishonest, malicious or very incompetent) labelers to further enhance the quality of crowd-sourcing. Extension to the case when a labeler's quality is task-type dependent is also discussed using techniques from the literature on continuous arms. We present numerical results using both simulation and a real dataset on a set of images labeled by Amazon Mechanic Turks (AMT).

OnTac: Online Task Assignment for Crowdsourcing

Task Assignment with Guaranteed Quality for Crowdsourcing Platforms.

Crowdota: An Online Task Assignment System In Crowdsourcing

Collaborative Task Allocation in Mobile Crowd Sensing.

Efficient Cross Dynamic Task Assignment in Spatial Crowdsourcing.

An Online Learning Approach to Improving the Quality of Crowd-Sourcing

Enhancing User Experience of Task Assignment in Spatial Crowdsourcing: A Self-Adaptive Batching Approach

Cross Online Assignment of Hybrid Task in Spatial Crowdsourcing

Quality-Assured Synchronized Task Assignment in Crowdsourcing

Online Spatial Crowdsensing With Expertise-Aware Truth Inference and Task Allocation

Bandit-Based Task Assignment for Heterogeneous Crowdsourcing

Efficient Online Crowdsourcing with Complex Annotations

Non-Rejection Aware Online Task Assignment in Spatial Crowdsourcing

Task Assignment for Social-Oriented Crowdsourcing.

Crowdsourced POI Labelling: Location-aware Result Inference and Task Assignment.

Environment-Driven Task Allocation in Heterogeneous Spatial Crowdsourcing

Batch-Based Cooperative Task Assignment in Spatial Crowdsourcing.

Using Differential Evolution To Estimate Labeler Quality For Crowdsourcing

An Online Task Assignment Based On Quality Constraint For Spatio-Temporal Crowdsourcing

Crowdsourcing System for Numerical Tasks Based on Latent Topic Aware Worker Reliability

Hierarchical Crowdsourcing for Data Labeling with Heterogeneous Crowd.