Abstract:We consider a crowd-sourcing problem where in the process of labeling massive datasets, multiple labelers with unknown annotation quality must be selected to perform the labeling task for each incoming data sample or task, with the results aggregated using for example simple or weighted majority voting rule. In this paper we approach this labeler selection problem in an online learning framework, whereby the quality of the labeling outcome by a specific set of labelers is estimated so that the learning algorithm over time learns to use the most effective combinations of labelers. This type of online learning in some sense falls under the family of multi-armed bandit (MAB) problems, but with a distinct feature not commonly seen: since the data is unlabeled to begin with and the labelers' quality is unknown, their labeling outcome (or reward in the MAB context) cannot be directly verified; it can only be estimated against the crowd and known probabilistically. We design an efficient online algorithm LS_OL using a simple majority voting rule that can differentiate high- and low-quality labelers over time, and is shown to have a regret (w.r.t. always using the optimal set of labelers) of O(log 2 T) uniformly in time under mild assumptions on the collective quality of the crowd, thus regret free in the average sense. We discuss performance improvement by using a more sophisticated majority voting rule, and show how to detect and filter out "bad" (dishonest, malicious or very incompetent) labelers to further enhance the quality of crowd-sourcing. Extension to the case when a labeler's quality is task-type dependent is also discussed using techniques from the literature on continuous arms. We present numerical results using both simulation and a real dataset on a set of images labeled by Amazon Mechanic Turks (AMT).

Label Aggregation with Self-Supervision Enhanced Graph Transformer

Learning from Crowds under Experts' Supervision

Automated Graph Self-supervised Learning via Multi-teacher Knowledge Distillation

NGAT: Attention in Breadth and Depth Exploration for Semi-Supervised Graph Representation Learning

A Formalized Framework for Incorporating Expert Labels in Crowdsourcing Environment

Hierarchical Crowdsourcing for Data Labeling with Heterogeneous Crowd.

Exploiting Heterogeneous Graph Neural Networks with Latent Worker/Task Correlation Information for Label Aggregation in Crowdsourcing

Active learning with confidence-based answers for crowdsourcing labeling tasks.

Truth Discovery in Sequence Labels from Crowds

Optimizing the Wisdom of the Crowd: Inference, Learning, and Teaching

Crowd-Certain: Label Aggregation in Crowdsourced and Ensemble Learning Classification

Active Learning For Crowdsourcing Using Knowledge Transfer

Label Consistency-Based Ground Truth Inference for Crowdsourcing

A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregation

Human-LLM Hybrid Text Answer Aggregation for Crowd Annotations

Self-Taught Active Learning from Crowds

Learning from Crowds Using Graph Neural Networks with Attention Mechanism

Adaptive Crowdsourcing Via Self-Supervised Learning

Labelling Training Samples Using Crowdsourcing Annotation for Recommendation

An Online Learning Approach to Improving the Quality of Crowd-Sourcing

Cost-efficient Crowdsourcing for Span-based Sequence Labeling: Worker Selection and Data Augmentation