Quality-Assured Synchronized Task Assignment in Crowdsourcing

Jiayang Tu,Peng Cheng,Lei Chen

DOI: https://doi.org/10.48550/arXiv.1806.00637

2018-06-02

Abstract:With the rapid development of crowdsourcing platforms that aggregate the intelligence of Internet workers, crowdsourcing has been widely utilized to address problems that require human cognitive abilities. Considering great dynamics of worker arrival and departure, it is of vital importance to design a task assignment scheme to adaptively select the most beneficial tasks for the available workers. In this paper, in order to make the most efficient utilization of the worker labor and balance the accuracy of answers and the overall latency, we a) develop a parameter estimation model that assists in estimating worker expertise, question easiness and answer confidence; b) propose a \textit{quality-assured synchronized task assignment scheme} that executes in batches and maximizes the number of potentially completed questions (MCQ) within each batch. We prove that MCQ problem is NP-hard and present two greedy approximation solutions to address the problem. The effectiveness and efficiency of the approximation solutions are further evaluated through extensive experiments on synthetic and real datasets. The experimental results show that the accuracy and the overall latency of the MCQ approaches outperform the existing online task assignment algorithms in the synchronized task assignment scenario.

Databases

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to improve the efficiency and quality of task assignment in crowdsourcing platforms. Specifically, the authors focus on how to design a task - assignment scheme that can adaptively select the most beneficial tasks in consideration of the high - dynamic nature of workers' arrival and departure, so as to achieve the most efficient utilization of workers' labor and control the overall latency while ensuring the accuracy of answers. To this end, they proposed a quality - assured synchronous task - assignment scheme, aiming to maximize the number of questions that can be completed (MCQ) in each batch and update model parameters in a batch - processing manner, thereby better balancing accuracy and latency.

The key challenges in the paper include:
1. **Efficient parameter - estimation model**: It is necessary to develop a model to estimate workers' professional levels, the difficulty levels of questions, and the confidence levels of answers. These parameters are crucial for optimizing task assignment. At the same time, this model needs to run fast enough to avoid causing excessive latency.
2. **Balancing accuracy and overall latency**: Improve the accuracy of answers by reasonably allocating the number of task repetitions, but at the same time avoid increasing unnecessary latency or cost.

To address these challenges, the authors took the following measures:
- Developed a parameter - estimation model. This model assumes that workers' professional levels and the difficulty levels of questions are two latent parameters that jointly determine the confidence level of answers. The model accelerates the estimation process through iterative calculation and adaptively updates parameters during the question - answering process without prior knowledge of workers' information.
- Proposed a quality - assured synchronous task - assignment scheme based on the above parameter - estimation model. This scheme runs in batches and updates model parameters after each task assignment. The purpose is to maximize the number of questions completed in each batch while minimizing the size of the task set assigned to workers, so as to concentrate workers' labor on the smallest set of questions, thereby controlling the overall latency and reserving more available questions for the next batch of workers.

In addition, the paper also proved that the MCQ problem is an NP - hard problem and proposed two greedy approximation algorithms to solve this problem. Through extensive experiments on synthetic and real - world data sets, the effectiveness and efficiency of these two algorithms were verified. The experimental results show that, compared with existing online task - assignment algorithms, the proposed MCQ method exhibits higher accuracy and lower overall latency in synchronous task - assignment scenarios.