Practical POMDP-based Test Mechanism for Quality Assurance in Volunteer Crowdsourcing.

Peng Shi,Wanyuan Wang,Yifeng Zhou,Jiuchuan Jiang,Yichuan Jiang,Zhifeng Hao,Jianyong Yu

DOI: https://doi.org/10.1080/17517575.2018.1493539

2018-01-01

Enterprise Information Systems

Abstract:In volunteer crowdsourcing, tasks are published via an open call and completed by many workers without reward. Under the traditional volunteer crowdsourcing paradigm, workers with diverse levels of reliabilities are chosen indiscriminately; moreover, each worker's performance may change over the time. Thus, the quality of task completions is a key concern in volunteer crowdsourcing. To improve the task completion quality (i.e. the accuracy of task answers), we adopt an adaptive test task (with a true answer) insertion approach to detect a worker's performance dynamically, thereby ensuring that normal tasks (with unknown true answers) are assigned when this worker is currently deemed reliable via testing. To decide when to route test tasks to detect a worker's performance or assign normal tasks to be completed in a high quality state, we proposed a Partially Observable Markov Decision Processes (POMDP) based test mechanism without any complicated parameter estimation, which is more practical for real-world volunteer crowdsourcing applications. In addition, we also designed rejection strategies to reject malicious workers and dubious answers. Experiments on real datasets demonstrate that the proposed test mechanism performs better in the accuracy of task answers, compared with benchmark methods.