Abstract:Collecting labels for data is important for many practical applications (e.g., data mining). However, this process can be expensive and time-consuming since it needs extensive efforts of domain experts. To decrease the cost, many recent works combine crowdsourcing, which outsources labeling tasks (usually in the form of questions) to a large group of non-expert workers, and active learning, which actively selects the best instances to be labeled, to acquire labeled datasets. However, for difficult tasks where workers are uncertain about their answers, asking for discrete labels might lead to poor performance due to the low-quality labels. In this paper, we design questions to get continuous worker responses which are more informative and contain workers’ labels as well as their confidence. As crowd workers may make mistakes, multiple workers are hired to answer each question. Then, we propose a new aggregation method to integrate the responses. By considering workers’ confidence information, the accuracy of integrated labels is improved. Furthermore, based on the new answers, we propose a novel active learning framework to iteratively select instances for “labeling”. We define a score function for instance selection by combining the uncertainty derived from the classifier model and the uncertainty derived from the answer sets. The uncertainty derived from uncertain answers is more effective than that derived from labels. We also propose batch methods which select multiple instances at a time to further improve the efficiency of our approach. Experimental studies on both simulated and real data show that our methods are effective in increasing the labeling accuracy and achieve significantly better performance than existing methods.

Exploiting predicted answer in label aggregation to make better use of the crowd wisdom

Attention-Aware Answers of the Crowd

Learning from Crowds under Experts' Supervision

Label Aggregation with Clustering for Biased Crowdsourced Labeling.

Active learning with confidence-based answers for crowdsourcing labeling tasks.

Optimizing the Wisdom of the Crowd: Inference, Learning, and Teaching

Learning True Labels from Noisy Crowd-Sourced Annotations

Label Aggregation with Self-Supervision Enhanced Graph Transformer

Hierarchical Crowdsourcing for Data Labeling with Heterogeneous Crowd.

A Subjectivity-Aware Algorithm for Label Aggregation in Crowdsourcing

Multi-label Answer Aggregation Based on Joint Matrix Factorization

Aggregating Crowd Wisdom with Side Information Via a Clustering-based Label-aware Autoencoder

Aggregating Crowd Wisdoms with Label-aware Autoencoders.

A Little Truth Injection but a Big Reward: Label Aggregation with Graph Neural Networks.

Crowd-Certain: Label Aggregation in Crowdsourced and Ensemble Learning Classification

Exploiting Heterogeneous Graph Neural Networks with Latent Worker/Task Correlation Information for Label Aggregation in Crowdsourcing

Truth Discovery in Sequence Labels from Crowds

Human-LLM Hybrid Text Answer Aggregation for Crowd Annotations

Temporal-aware Language Representation Learning from Crowdsourced Labels

Aggregating Crowd Wisdom With Instance Grouping Methods

Interactive Learning with Proactive Cognition Enhancement for Crowd Workers.