Abstract:Crowd-selection is essential to crowdsourcing applications, since choosing the right workers with particular expertise to carry out specific crowdsourced tasks is extremely important. The central problem is simple but tricky: given a crowdsourced task, is the right worker to ask? Currently, most existing work has mainly studied the problem of crowd-selection for simple crowdsourced tasks such as decision making and sentiment analysis. Their crowd- selection procedures are based on the trustworthiness of workers. However, for some complex tasks such as document review and question answering, selecting workers based on the latent category of tasks is a better solution. In this paper, we formulate a new problem of task-driven crowd- selection for complex tasks. We first develop a bayesian gener- ative model to exploit who knows what for the workers in the crowdsourcing environment. The model provides a principle and natural framework for capturing the latent skills of workers as well as the latent categories of crowdsourced tasks. The inference of the latent skills of workers is based on past resolved crowdsourced tasks with feedback scores. We assume that the feedback scores can illustrate the performance of the workers for the tasks. We then devise a variational algorithm that transforms the latent skill infer- ence with the proposed model into a standard optimization prob- lem, which can be solved efficiently. We verify the performance of our method through extensive experiments on the data collected from three well-known crowdsourcing platforms for question an- swering tasks such as Quora, Yahoo ! Answer and Stack Overflow.

The Effect of Class Imbalance and Order on Crowdsourced Relevance Judgments

Task Assignment with Guaranteed Quality for Crowdsourcing Platforms.

Learning from Crowds under Experts' Supervision

Recovering Missing Labels of Crowdsourcing Workers.

The Importance of Being Earnest in Crowdsourcing Systems

Learning from Crowds in the Presence of Schools of Thought.

Listwise Approach For Rank Aggregation In Crowdsourcing

Measuring Social Biases of Crowd Workers using Counterfactual Queries

Crowd-Selection Query Processing in Crowdsourcing Databases: A Task-Driven Approach.

Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collections Accurately and Affordably

Avoiding Imposters and Delinquents: Adversarial Crowdsourcing and Peer Prediction

Efficiency of active learning for the allocation of workers on crowdsourced classification tasks

Similarity-driven and task-driven models for diversity of opinion in crowdsourcing markets

Crowdsourcing Label Quality: A Theoretical Analysis

Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learning

Multicategory Crowdsourcing Accounting for Plurality in Worker Skill and Intention, Task Difficulty, and Task Heterogeneity

Hear Me Out: A Study on the Use of the Voice Modality for Crowdsourced Relevance Assessments

Quantifying the Impact of Cognitive Biases in Question-Answering Systems

Optimal Stopping and Worker Selection in Crowdsourcing: an Adaptive Sequential Probability Ratio Test Framework

False Discovery Rate Control and Statistical Quality Assessment of Annotators in Crowdsourced Ranking

Collusion-Proof Result Inference in Crowdsourcing