Data Dialogue with ChatGPT: Using Code Interpreter to Simulate and Analyse Experimental Data

Andrew Low,Z. Yasemin Kalender

2024-01-01

Abstract:Artificial Intelligence (AI) has the potential to fundamentally change the educational landscape. So far, much of the physics education research relating to AI has focused on lecture-based assessment and the ability of ChatGPT to answer conceptual surveys and traditional exam-style questions. In this study, we shift the focus by investigating ChatGPT's ability to complete an introductory mechanics laboratory activity by using Code Interpreter, a recent plugin that allows users to generate and analyse data by writing and running Python code `behind the scenes'. By uploading a common `spring constant' lab activity using Code Interpreter, we investigate the ability of ChatGPT to interpret the activity, generate realistic model data, produce a line-fit, and calculate the reduced chi square statistic. By analysing our interactions with ChatGPT, along with the Python code generated by Code Interpreter, we assess how the quality and accuracy of ChatGPT's responses depends on different levels of prompt detail. We find that although ChatGPT is capable of completing the lab activity and generating plausible-looking data, the quality of the output is highly dependent on the detail and specificity of the text prompts provided. We find that the data generation process adopted by ChatGPT in this study leads to heteroscedasticity in the simulated data, which may be difficult for novice learners to spot. We also find that when real experimental data is uploaded via Code Interpreter, ChatGPT is capable of correctly plotting and fitting the data, calculating the spring constant and associated uncertainty, and calculating the reduced chi square statistic. This work offers new insights into the capabilities of Code Interpreter within a laboratory setting and highlights a variety of text-prompt strategies for the effective use of Code Interpreter in a lab context.

Physics Education

What problem does this paper attempt to address?

The paper primarily explores how to utilize ChatGPT in combination with the Code Interpreter plugin to accomplish data analysis tasks in physics experiments and evaluates the potential application of this combination in educational settings.

The core issue of the paper is to test the capabilities of the Code Interpreter plugin, particularly whether it can effectively interpret and complete a basic physics laboratory activity—the spring constant measurement experiment. Specifically, the study focuses on the following three key questions:

1. **Data Generation and Analysis Capability**: Can ChatGPT combined with the Code Interpreter plugin successfully generate simulated data, perform data analysis, and interpret data related to basic laboratory activities?

- Researchers tested ChatGPT's data generation and analysis capabilities through different prompting strategies. The results indicate that although ChatGPT can complete experimental activities and generate seemingly reasonable data, the quality of the output is highly dependent on the level of detail provided in the text prompts.

2. **Effective Communication Strategies**: How can interactions with ChatGPT be optimized to ensure high-quality and reliable outputs?

- Researchers found that detailed prompts can significantly improve the quality of ChatGPT's responses. For example, in the second method, after providing detailed instructions, ChatGPT was able to correctly simulate data, plot graphs, and conduct reasonable error propagation analysis.

3. **Impact on Laboratory Courses**: How do AI tools like ChatGPT affect introductory laboratory courses?

- The study shows that ChatGPT combined with the Code Interpreter can provide valuable support in a laboratory environment, but it may also present some challenges, such as the generated data potentially exhibiting heteroscedasticity, which may be difficult for beginners to identify.

Through the above research, the paper aims to provide insights for educators, researchers, and policymakers on how to adapt AI technology in laboratory science education. Additionally, the study emphasizes the importance of fostering critical thinking, especially when students use AI technology.