Abstract:Recently, crowdsourcing has established itself as an efficient labeling solution by distributing tasks to crowd workers. As the workers can make mistakes with diverse expertise, one core learning task is to estimate each worker's expertise , and aggregate over them to infer the latent true labels . In this paper, we show that as one of the major research directions, the noise transition matrix based worker expertise modeling methods commonly overfit the annotation noise, either due to the oversimplified noise assumption or inaccurate estimation. To solve this problem, we propose a knowledge distillation framework (KD-Crowd) by combining the complementary strength of noise-model-free robust learning techniques and transition matrix based worker expertise modeling. The framework consists of two stages: in Stage 1, a noise-model-free robust student model is trained by treating the prediction of a transition matrix based crowdsourcing teacher model as noisy labels, aiming at correcting the teacher's mistakes and obtaining better true label predictions; in Stage 2, we switch their roles, retraining a better crowdsourcing model using the crowds' annotations supervised by the refined true label predictions given by Stage 1. Additionally, we propose one f-mutual information gain ( MIG f ) based knowledge distillation loss, which finds the maximum information intersection between the student's and teacher's prediction. We show in experiments that MIG f achieves obvious improvements compared to the regular KL divergence knowledge distillation loss, which tends to force the student to memorize all information of the teacher's prediction, including its errors. We conduct extensive experiments showing that, as a universal framework, KD-Crowd substantially improves previous crowdsourcing methods on true label prediction and worker expertise estimation.

Extracting the Wisdom of a Smaller Crowd Fromdependent Quantile Judgments

A Formalized Framework for Incorporating Expert Labels in Crowdsourcing Environment

Wisdom of the crowd from unsupervised dimension reduction

Using Cognitive Models to Improve the Wisdom of the Crowd

Learning to Predict the Wisdom of Crowds

The Wisdom of Reluctant Crowds

When is a crowd wise?

Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds

Wisdom of the Confident: Using Social Interactions to Eliminate the Bias in Wisdom of the Crowds

Learning from Crowds in the Presence of Schools of Thought.

KD-Crowd: a knowledge distillation framework for learning from crowds

Elicitation and aggregation of multimodal estimates improve wisdom of crowd effects on ordering tasks

Crowd IQ -- Aggregating Opinions to Boost Performance

How social influence can undermine the wisdom of crowd effect

Adaptive Crowdsourcing Via Self-Supervised Learning

Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy

Unsupervised Opinion Aggregation -- A Statistical Perspective

Value and Misinformation in Collaborative Investing Platforms.

Wisdom of the Crowd Voting: Truthful Aggregation of Voter Information and Preferences

Crowd prediction systems: Markets, polls, and elite forecasters

High-Dimensional Crowdsourced Data Distribution Estimation with Local Privacy.