Hidden Markov Based Truth Discovery for Multi-Agent Labeling.

Shanyang Jiang,Lan Zhang
DOI: https://doi.org/10.1109/bigcom53800.2021.00038
2021-01-01
Abstract:In recent years, large-scale data labeling has become a huge demand and extremely challenging task. More and more machine learning and deep learning methods have been proposed to generate a variety of semantic labels for data, and used to provide data labeling service as a data labeling agent. However, even todays machine learning and deep learning model may also output a wrong label. So it is necessary to optimize the quality of the collected labeling results. Generally, there are many labeling models in the data labeling market, and each label agent has a different degree of reliability. Service buyers also need to integrate many label answers from different labels agents to get the final correct label truth. In this article, we design a novel probabilistic graph based truth discovery algorithm to estimate the true label truth of the target task and the reliability of the label agent through the collected label answers. In particular, we build a novel expectation-maximization based iterative method for truth discovery to inference label truth and estimate label agent reliability. Finally, we conduct several experiments on a real-world dataset to testify the performance of our method.
What problem does this paper attempt to address?