Student and AI responses to physics problems examined through the lenses of sensemaking and mechanistic reasoning

Amogh Sirnoorkar,Dean Zollman,James T. Laverty,Alejandra J. Magana,Sanjay Rebello,Lynn A. Bryan

2024-01-01

Abstract:Several reports in education have called for transforming physics learning environments by promoting sensemaking of real-world scenarios in light of curricular ideas. Recent advancements in Generative-Artificial Intelligence has garnered increasing traction in educators' community by virtue of its potential in transforming STEM learning. In this exploratory study, we adopt a mixed-methods approach in comparatively examining student- and AI-generated responses to two different formats of a physics problem through the cognitive lenses of sensemaking and mechanistic reasoning. The student data is derived from think-aloud interviews of introductory students and the AI data comes from ChatGPT's solutions collected using Zero shot approach. The results highlight AI responses to evidence most features of the two processes through well-structured solutions and student responses to effectively leverage representations in their solutions through iterative refinement of arguments. In other words, while AI responses reflect how physics is talked about, the student responses reflect how physics is practiced. Implications of these results in light of development and deployment of AI systems in physics pedagogy are discussed.

Physics Education

What problem does this paper attempt to address?

This paper explores the differences between artificial intelligence (AI), particularly ChatGPT, and student answers when solving physics problems from a cognitive perspective - sensemaking and mechanistic reasoning. The study used a mixed-methods approach to analyze data from student think-aloud interviews and ChatGPT's unprompted responses. The results indicate that AI's answers exhibit features of sensemaking and mechanistic reasoning, even in incorrect inferences. In contrast, student answers reflect more practical physics practices, such as iterative reasoning and chart-based reasoning. The paper emphasizes that AI answers reflect how physics is discussed, whereas student answers reflect how physics is practiced. The study discusses the implications of these results for designing classroom activities involving generative AI and identifies directions for future research.