Abstract:Crowd-selection is essential to crowdsourcing applications, since choosing the right workers with particular expertise to carry out specific crowdsourced tasks is extremely important. The central problem is simple but tricky: given a crowdsourced task, is the right worker to ask? Currently, most existing work has mainly studied the problem of crowd-selection for simple crowdsourced tasks such as decision making and sentiment analysis. Their crowd- selection procedures are based on the trustworthiness of workers. However, for some complex tasks such as document review and question answering, selecting workers based on the latent category of tasks is a better solution. In this paper, we formulate a new problem of task-driven crowd- selection for complex tasks. We first develop a bayesian gener- ative model to exploit who knows what for the workers in the crowdsourcing environment. The model provides a principle and natural framework for capturing the latent skills of workers as well as the latent categories of crowdsourced tasks. The inference of the latent skills of workers is based on past resolved crowdsourced tasks with feedback scores. We assume that the feedback scores can illustrate the performance of the workers for the tasks. We then devise a variational algorithm that transforms the latent skill infer- ence with the proposed model into a standard optimization prob- lem, which can be solved efficiently. We verify the performance of our method through extensive experiments on the data collected from three well-known crowdsourcing platforms for question an- swering tasks such as Quora, Yahoo ! Answer and Stack Overflow.

Optimal Stopping and Worker Selection in Crowdsourcing: an Adaptive Sequential Probability Ratio Test Framework

Stable Job Assignment for Crowdsourcing.

Statistical Decision Making for Optimal Budget Allocation in Crowd Labeling

Dynamic Task Allocation for Crowdsourcing Settings

Bayesian Decision Process for Cost-Efficient Dynamic Ranking via Crowdsourcing

Toward optimal allocation of location dependent tasks in crowdsensing

An online cost sensitive decision-making method in crowdsourcing systems.

Crowdsourcing with Unsure Option

Bayesian Decision Process for Budget-efficient Crowdsourced Clustering

Best of Both Worlds: Mitigating Imbalance of Crowd Worker Strategic Choices Without a Budget

Effective Solution for Labeling Candidates with a Proper Ration for Efficient Crowdsourcing

Crowd-Selection Query Processing in Crowdsourcing Databases: A Task-Driven Approach.

Cost-efficient Crowdsourcing for Span-based Sequence Labeling: Worker Selection and Data Augmentation

Incentivize Crowd Labeling Under Budget Constraint

Incentivize Multi-Class Crowd Labeling Under Budget Constraint.

Bayes-Optimal Effort Allocation in Crowdsourcing: Bounds and Index Policies

Robust Cumulative Crowdsourcing Framework Using New Incentive Payment Function and Joint Aggregation Model.

Multi-Objective Crowd Worker Selection in Crowdsourced Testing

Attention-Aware Answers of the Crowd

Optimizing Crowdsourcing Task Assignment Policies Using Multi-Agent Reinforcement Learning in Stochastic Games

Theoretical Guarantee for Crowdsourcing Learning with Unsure Option