StuGPTViz: A Visual Analytics Approach to Understand Student-ChatGPT Interactions

Zixin Chen,Jiachen Wang,Meng Xia,Kento Shigyo,Dingdong Liu,Rong Zhang,Huamin Qu

2024-09-17

Abstract:The integration of Large Language Models (LLMs), especially ChatGPT, into education is poised to revolutionize students' learning experiences by introducing innovative conversational learning methodologies. To empower students to fully leverage the capabilities of ChatGPT in educational scenarios, understanding students' interaction patterns with ChatGPT is crucial for instructors. However, this endeavor is challenging due to the absence of datasets focused on student-ChatGPT conversations and the complexities in identifying and analyzing the evolutional interaction patterns within conversations. To address these challenges, we collected conversational data from 48 students interacting with ChatGPT in a master's level data visualization course over one semester. We then developed a coding scheme, grounded in the literature on cognitive levels and thematic analysis, to categorize students' interaction patterns with ChatGPT. Furthermore, we present a visual analytics system, StuGPTViz, that tracks and compares temporal patterns in student prompts and the quality of ChatGPT's responses at multiple scales, revealing significant pedagogical insights for instructors. We validated the system's effectiveness through expert interviews with six data visualization instructors and three case studies. The results confirmed StuGPTViz's capacity to enhance educators' insights into the pedagogical value of ChatGPT. We also discussed the potential research opportunities of applying visual analytics in education and developing AI-driven personalized learning solutions.

Human-Computer Interaction,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the issue of how to enhance educational outcomes by analyzing the interactions between students and large language models (such as ChatGPT). Specifically, the paper focuses on the following aspects:

1. **Lack of Datasets**: Currently, there is a lack of datasets specifically targeting interactions between students and ChatGPT. Existing datasets mainly contain everyday conversations of ordinary users, and the quality of these conversations cannot be guaranteed. Moreover, most of them adopt a "question-and-answer" format, making it difficult to reflect real educational scenarios.

2. **Understanding Interaction Patterns**: Teachers need to understand students' cognitive levels and thinking patterns when using these advanced AI tools. However, existing research methods find it challenging to accurately interpret students' cognitive levels and usage capabilities.

3. **Visualization Analysis Tools**: Existing visualization tools mostly focus on various aspects of dialogue analysis, such as sentiment analysis and topic modeling, but lack in-depth analysis of one-on-one interactions between students and AI tools, especially in capturing changes in cognitive levels.

To address the above challenges, the researchers conducted the following work:
- Introduced ChatGPT in a graduate-level data visualization course and collected a large amount of high-quality interaction data between students and ChatGPT.
- Developed a coding scheme based on literature review and thematic analysis to classify students' interaction patterns.
- Designed a visualization analysis system named StuGPTViz to help teachers explore the interaction patterns between students and ChatGPT from multiple perspectives, thereby gaining valuable educational insights.
- Validated the effectiveness of the system through expert interviews and case studies, and explored research opportunities for applying visualization analysis in education and developing personalized learning solutions.

In summary, this paper provides a new approach to understanding and analyzing the interactions between students and ChatGPT, offering teachers deeper educational insights.