Real-time quality control for crowdsourcing relevance evaluation

Tao Xia,Chuang Zhang,Jingjing Xie,Tai Li
DOI: https://doi.org/10.1109/ICNIDC.2012.6418811
2012-01-01
Abstract:In recent years, crowdsourcing has become an effective method in many fields, such as relevance evaluation of search engine results. However, the problems of bad workers and quality control always stand among the major challenges within the realm. Based on our experiments carried out in TREC 2011 Crowdsourcing track, this paper demonstrates a realtime strategy in recruiting workers and monitoring the quality of their relevance and rank judgments. The effectiveness of our strategy has been verified by empirical results.
What problem does this paper attempt to address?