Abstract:We consider a crowd-sourcing problem where in the process of labeling massive datasets, multiple labelers with unknown annotation quality must be selected to perform the labeling task for each incoming data sample or task, with the results aggregated using for example simple or weighted majority voting rule. In this paper we approach this labeler selection problem in an online learning framework, whereby the quality of the labeling outcome by a specific set of labelers is estimated so that the learning algorithm over time learns to use the most effective combinations of labelers. This type of online learning in some sense falls under the family of multi-armed bandit (MAB) problems, but with a distinct feature not commonly seen: since the data is unlabeled to begin with and the labelers' quality is unknown, their labeling outcome (or reward in the MAB context) cannot be directly verified; it can only be estimated against the crowd and known probabilistically. We design an efficient online algorithm LS_OL using a simple majority voting rule that can differentiate high- and low-quality labelers over time, and is shown to have a regret (w.r.t. always using the optimal set of labelers) of O(log 2 T) uniformly in time under mild assumptions on the collective quality of the crowd, thus regret free in the average sense. We discuss performance improvement by using a more sophisticated majority voting rule, and show how to detect and filter out "bad" (dishonest, malicious or very incompetent) labelers to further enhance the quality of crowd-sourcing. Extension to the case when a labeler's quality is task-type dependent is also discussed using techniques from the literature on continuous arms. We present numerical results using both simulation and a real dataset on a set of images labeled by Amazon Mechanic Turks (AMT).

Learning to Predict the Wisdom of Crowds

Recovering Missing Labels of Crowdsourcing Workers.

Learning from Crowds under Experts' Supervision

Adaptive Crowdsourcing Via Self-Supervised Learning

Learning from Crowds in the Presence of Schools of Thought.

When is a crowd wise?

Wisdom of the crowd from unsupervised dimension reduction

Unsupervised Crowdsourcing with Accuracy and Cost Guarantees

Extracting the Wisdom of a Smaller Crowd Fromdependent Quantile Judgments

Using Cognitive Models to Improve the Wisdom of the Crowd

Crowdsourcing Control: Moving Beyond Multiple Choice

An Online Learning Approach to Improving the Quality of Crowd-Sourcing

Selective Verification Strategy for Learning from Crowds

Attention-Aware Answers of the Crowd

Optimizing the Wisdom of the Crowd: Inference, Learning, and Teaching

Avoiding Imposters and Delinquents: Adversarial Crowdsourcing and Peer Prediction

Wisdom of Crowds Algorithm for Stock Market Predictions

Metawisdom of the Crowd: How Choice Within Aided Decision Making Can Make Crowd Wisdom Robust

Crowdsourcing with Unsure Option

Adaptive Algorithms for Crowd-Aided Categorization

Crowdsourcing in the Absence of Ground Truth -- A Case Study