Leveraging the power of generative AI: a case study on feedback analysis of student evaluation in an undergraduate physiology practical course

Angelina Fong,Julian Lin,Lauren Ryan,Melody Chang

DOI: https://doi.org/10.1152/physiol.2024.39.s1.2081

IF: 11.8461

2024-05-01

Physiology

Abstract:Student surveys with Likert scales and open responses are key to gauging the student experience in educational institutions. However, the thematic analysis of open responses is time-consuming, delaying feedback. This study aims to evaluate the effcacy of ChatGPT-4, a generative AI large language model (LLM) to streamline thematic analysis of student perception surveys. We hypothesise that LLMs can expedite the process, however, human intervention remains essential. The study focused on a 2nd-year physiology course’s and evaluated comparing online vs face-to-face (F2F) delivery, to determine if practical classes could successfully be delivered to students online without compromising the delivery of the desired skills and learning outcomes. Data from six cohorts were included (2019-2022); three semesters online and three F2F. Overall grades, and grades from individual written assessments requiring data analysis and critical thinking showed no difference between the different delivery modes, indicating that major learning outcomes are maintained in online delivery. Student perception was analysed from an online cohort (Semester 2, 2022). Analysis of the Likert data from the student survey from an online cohort (response rate: 40/202) found that students strongly agreed that the class was enjoyable (83% agreement) and the online tools and software were easy to use (83% agreement). Thematic analysis was performed on the open text responses using a LLM (ChatGPT-4) guided by a structured thematic analysis framework and was conducted in three phases: coding responses, collating codes into themes, and visualizing these themes. Each phase required precise prompt engineering to ensure the outputs were accurate and relevant. Thematic analysis using ChatGPT-4 identified that students enjoyed the social aspects of the teamwork and collaboration. The students found the online and software tools easy to use due to rapid feedback from instructors. Altogether, this produced a positive experience in their online learning experiments. A significant advantage to using ChatGPT-4 is the rapid processing of the thematic analysis and alleviate the burdensome aspects of qualitative analysis. Thus, allowing for the timely extraction of nuanced findings provided by qualitative data, and ensuring that student feedback can be effectively addressed. While the results showed that ChatGPT-4 was largely successful in processing the qualitative data, human oversight was necessary to correct minor errors and ensure logical consistency. In addition, LLMs like ChatGPT-4 cannot operate in isolation; human involvement are imperative in making evaluative judgments and checking for hallucinations. Nevertheless, we present a framework for a collaborative human-LLM approach to qualitative analysis of student evaluations to provide more timely feedback and action. The increase rapidity in feedback will help alleviate the student believe that their feedback goes unread and unheeded, thereby improving student outcomes. This is the full abstract presented at the American Physiology Summit 2024 meeting and is only available in HTML format. There are no additional versions or additional content available for this abstract. Physiology was not involved in the peer review process.

physiology