The Effect of Individual-Level Factors and Task Features on Interface Design for Rule-Verification Crowdsourcing Tasks
Wen WuXixiong ZhouXiaowen ShiPing WuXin LinJing YangLiang Hea School of Computer Science and Technology,East China Normal University,Shanghai,Chinab Shanghai Key Laboratory of Mental Health and Psychological Crisis Intervention,School of Psychology and Cognitive Science,East China Normal University,Shanghai,ChinaWen Wu is an associate professor in the School of Computer Science and Technology at East China Normal University. Her primary interests lie in the areas of user modeling,recommender systems,HCI,and AI. She has awarded Best Journal Paper in UMUAI'19 and Best Student Paper in UMAP'15.Xixiong Zhou is a master in Computer Science at East China Normal University. His research mainly explores the human computer interaction,user modeling,and crowdsourcing.Xiaowen Shi is a master in Computer Science at East China Normal University. Her research interests include information extraction,design science,and human computer interaction.Ping Wu is a master in Computer Science at East China Normal University. His work focuses on knowledge graph,knowledge extraction,and knowledge discovering.Xin Lin received the BE degree and the PhD degree,both in computer science and engineering,from Zhejiang University,Hangzhou,China. He is currently a professor at the Department of Computer Science,East China Normal University,Shanghai,China. His current research interests include knowledge graph and large language models.Jing Yang is an associate professor in the School of Computer Science and Technology at East China Normal University. Her main research interests include the construction of common-sense knowledge bases,crowdsourcing task allocation,and the application of humans in the loop.Liang He is a professor and the vice dean of the School of Computer Science and Technology,East China Normal University. His current research interest includes natural language processing,knowledge processing,and human-machine hybrid enhanced intelligence. He has published articles in top conferences and journals like SIGIR and TKDE.
DOI: https://doi.org/10.1080/10447318.2024.2332031
IF: 4.92
2024-04-17
International Journal of Human-Computer Interaction
Abstract:Rule-verification is essential for the construction of high-quality knowledge bases. Compared with the traditional approach of inviting experts with domain knowledge to accomplish the rule-verification task, crowdsourcing is a more efficient way, which could save time and effort. However, the current presentation mode of rules relies primarily on predicate logic. On one hand, predicate logic is rather abstract and complex, making it difficult for ordinary users without domain knowledge to comprehend. On the other hand, the singular presentation mode lacks personalization, failing to provide targeted assistance to different users. In this work, to facilitate users' understanding of rules, we are motivated to design three explainable interfaces by incorporating the information presentation modes including instance (INS), natural language (NL), and knowledge graph (KG). Through our first-round quasi-experimental study, we find that NL and KG interfaces are preferred by users in general for the rule-verification crowdsourcing tasks. More notably, we identify the important role of users' personality, cognitive style, and tasks' difficulty level in influencing users' perception of the interface design. For instance, among those who score low on OpennesstoExperience, verbalizers perceive NL interface more useful than visualizers . In addition, users are more likely to use KG interface when dealing with some complicated tasks. Inspired by our findings, we further develop a personalized interface recommendation model with the consideration of individual-level factors and tasks' features. According to the results of both modeling comparison and the second-round user evaluation, we observe that recommending personalized interface based on our designed model could not only enhance users' subjective perception like perceived usefulness and satisfaction but also improve their task performance.
computer science, cybernetics,ergonomics