Crowdota: An Online Task Assignment System In Crowdsourcing
Xiang Yu,Guoliang Li,Yudian Zheng,Yan Huang,Songfan Zhang,Fei Chen
DOI: https://doi.org/10.1109/ICDE.2018.00193
2018-01-01
Abstract:Crowdsourcing is widely accepted as a means for resolving tasks that are hard for computers, e.g., entity resolution. Unfortunately, Crowdsourcing may yield relatively low-quality results if there is no proper quality control. Although previous studies attempt to eliminate workers by estimating workers' qualities via qualification tests or hidden tests, the qualities estimated may not be accurate, because workers may have diverse qualities across tasks. Thus, the quality of the results could be further improved by wisely assigning tasks to the workers who are specialized in the tasks and online task assignment is an effective way to achieve this goal. However, existing crowdsourcing platforms either do not support online task assignment (e.g., CrowdFlower) or are not user-friendly because they require to write complicated codes (e.g., Amazon MTurk). To address these limitations, we develop an online task assignment system, which can on-the-fly assign workers with appropriate tasks. We have deployed our system on top of MTurk. We demonstrate the following scenarios using our system. Firstly, requesters can easily utilize our system to enable online task assignment in order to improve answer quality. Moreover, requesters do not need to write any code. Secondly, our system can infer the quality of workers, and requesters can design and test their own task assignment algorithms using our proposed information. Thirdly, our system can monitor tasks and workers in real time, and the requesters can eliminate bad workers or terminate the crowdsourcing process to reduce the unnecessary cost.