CoQuest: Exploring Research Question Co-Creation with an LLM-based Agent

Yiren Liu,Si Chen,Haocong Cheng,Mengxia Yu,Xiao Ran,Andrew Mo,Yiliu Tang,Yun Huang

DOI: https://doi.org/10.48550/arXiv.2310.06155

2024-03-21

Abstract:Developing novel research questions (RQs) often requires extensive literature reviews, especially in interdisciplinary fields. To support RQ development through human-AI co-creation, we leveraged Large Language Models (LLMs) to build an LLM-based agent system named CoQuest. We conducted an experiment with 20 HCI researchers to examine the impact of two interaction designs: breadth-first and depth-first RQ generation. The findings revealed that participants perceived the breadth-first approach as more creative and trustworthy upon task completion. Conversely, during the task, participants considered the depth-first generated RQs as more creative. Additionally, we discovered that AI processing delays allowed users to reflect on multiple RQs simultaneously, leading to a higher quantity of generated RQs and an enhanced sense of control. Our work makes both theoretical and practical contributions by proposing and evaluating a mental model for human-AI co-creation of RQs. We also address potential ethical issues, such as biases and over-reliance on AI, advocating for using the system to improve human research creativity rather than automating scientific inquiry.

Human-Computer Interaction,Computational Engineering, Finance, and Science

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to develop novel research questions (RQs) through human - AI co - creation, especially in interdisciplinary research. Specifically, the paper explores the impact of different interaction designs (breadth - first and depth - first generation) of the artificial intelligence agent system (CoQuest) built with large language models (LLMs) when supporting researchers in generating research questions. The paper aims to understand how these different interaction designs affect users' experiences (such as perceived creativity, sense of trust and control over AI) and the quality of the generated research questions (such as creativity scores). In addition, the paper also explores some behavioral factors in the process of users interacting with AI, which are related to users' enhanced experiences and outcomes in human - AI co - creation. Through this research, the author hopes to make theoretical and practical contributions to human - AI co - creation of research questions and discuss the implications of designing ethical human - AI co - creation systems.