Manuel G. Forero,H. J. Herrera-Suárez
Abstract:This study investigates the influence of ChatGPT, an AI-based language model, on student performance in a physics course. We conducted an experimental analysis with two cohorts of students in a second-semester engineering physics course. The control group (Physics 2 2022B) used traditional teaching methods, while the experimental group (Physics 2 2023A) integrated ChatGPT as a learning tool. Our results indicate that the use of ChatGPT led to a significant decrease in student performance, as evidenced by lower grades and negative Hake factors compared to the control group. In addition, a survey of students revealed conflicting perceptions of the usefulness of ChatGPT in teaching physics. While most recognized its usefulness in understanding concepts and providing information, concerns were raised about its potential to reduce critical thinking and independent learning. These findings suggest that while ChatGPT can be a useful tool, it should be used with caution and as a supplement to traditional teaching methods, rather than as a stand-alone solution. The study underlines the importance of critical and reflective use of AI tools in educational settings and highlights the irreplaceable role of teachers in providing comprehensive educational support.
What problem does this paper attempt to address?
### Problems the Paper Attempts to Solve
This paper aims to explore the impact of the artificial intelligence language model ChatGPT on students' academic performance in physics courses. Specifically, the researchers experimentally analyzed the performance of two student groups: one group (control group) used traditional teaching methods, while the other group (experimental group) used ChatGPT as an auxiliary learning tool. The main goal of the study is to assess whether the use of ChatGPT has a positive or negative impact on students' academic performance.
### Research Background and Motivation
With the development of artificial intelligence technology, especially the emergence of large language models such as Generative Pre-trained Transformers (GPT), the application of these tools in the field of education has attracted widespread attention. However, there is still relatively little research on the specific impact of these tools on student learning and performance. Therefore, conducting such research to evaluate the actual effectiveness of AI tools like ChatGPT in education is particularly important.
### Research Methods
1. **Subjects**:
- **Control Group**: 40 students from the Physics 2 course in the 2022B semester, using traditional teaching methods.
- **Experimental Group**: 36 students from the Physics 2 course in the 2023A semester, using ChatGPT as an auxiliary learning tool.
2. **Experimental Design**:
- **Course Content**: Covers basic physics concepts such as vibrations, fluids, and thermodynamics.
- **Assessment Methods**: Includes 3 exams, each accounting for 20% of the total grade.
- **Teaching Methods**: The control group used traditional lectures, experiments, and assignments; the experimental group, in addition to traditional methods, was encouraged to use ChatGPT for learning and problem-solving.
3. **Data Collection**:
- **Grade Comparison**: Recorded the exam scores of both groups and calculated the average scores and Hake factor (used to assess the gain in conceptual understanding).
- **Questionnaire Survey**: Conducted a questionnaire survey of 41 physics students to understand their views and experiences with using ChatGPT.
### Main Results
1. **Grade Comparison**:
- The average scores of the experimental group were significantly lower than those of the control group, especially in various exam stages and overall performance.
- The Hake factor showed that the gain in conceptual understanding for the experimental group was negative, indicating a decline in their conceptual understanding ability.
2. **Questionnaire Survey Results**:
- Most students found ChatGPT helpful in answering physics-related questions, clarifying doubts, and providing relevant examples.
- However, a considerable number of students reported that using ChatGPT reduced their ability to think independently and critically.
- Students gave high ratings for ChatGPT's adaptability and timely responses but had some concerns about its accuracy.
### Conclusion
1. **Positive Impact of ChatGPT**:
- As an auxiliary learning tool, ChatGPT excels in answering questions, clarifying doubts, and providing relevant examples.
- For students needing help understanding physics concepts, ChatGPT is a valuable resource.
2. **Potential Negative Impact of ChatGPT**:
- Using ChatGPT may lead to a reduction in students' ability to think independently and critically.
- Some students expressed dissatisfaction with ChatGPT's accuracy and explanatory ability.
3. **Recommendations**:
- Educators should be cautious when considering the use of AI tools like ChatGPT, ensuring these tools do not replace the role of teachers.
- These tools should be used reflectively and critically to ensure they promote meaningful learning and the development of student skills.
In summary, this study provides valuable insights, revealing the potential advantages and limitations of ChatGPT in physics courses, and emphasizes the importance of the judicious use of AI tools in education.