Exploring the Potential of ChatGPT-4 for Clinical Decision Support in Cardiac Electrophysiology and Its Semi-Automatic Evaluation Metrics

Xiarepati Tieliwaerdi,Abulikemu Abuduweili,Saleh Saleh,Erasmus Mutabi,Michael A Rosenberg,Emerson Liu

DOI: https://doi.org/10.1101/2024.07.10.24310247

2024-07-12

Abstract:Background/Aim: Despite extensive research in other medical fields, the capabilities of ChatGPT-4 in clinical decision support within cardiac electrophysiology (EP) remain largely unexplored. This study aims to enhance ChatGPT- 4`s domain-specific expertise by employing the Retrieval-Augmented Generation (RAG) approach, which integrates up-to-date, evidence-based knowledge into ChatGPT-4`s foundational database. Additionally, we plan to explore the use of commonly used automatic evaluation metrics in natural language processing, such as BERTScore, BLEURT, and cosine similarity, alongside human evaluation, to develop a semi-automatic framework. This aims to reduce dependency on exhaustive human evaluations, addressing the need for efficient and scalable assessment tools in medical decision-making, given the rapid adoption of ChatGPT-4 by the public. Method: We analyzed five atrial fibrillation (Afib) cases and seven cardiac implantable electronic device (CIED) infection cases curated from PubMed case reports. We conducted a total of 120 experiments for Afib and 168 for CIED cases, testing each case across four temperature settings (0, 0.5, 1, 1.2) and three seed settings (1, 2, 3). ChatGPT-4`s performance was assessed under two modes: the Retrieval-Augmented Generation (RAG) mode and the Cold Turkey mode, which queries ChatGPT without external knowledge via RAG. For Afib cases, ChatGPT was asked to determine rate, rhythm, and anticoagulation options, and provide reasoning for each. For CIED cases, ChatGPT is asked to determine the presence of device infections. Accuracy metrics evaluated the determination component, while reasoning was assessed by human evaluation, BERTScore, BLEURT, and cosine similarity. A mixed effects analysis was used to compare the performance under both models across varying seeds and temperatures. Spearman`s rank correlation was used to explore the relationship between automatic metrics and human evaluation. Results: In this study, 120 experiments for Afib and 168 for CIED were conducted. There is no significant difference between the RAG mode and the Cold Turkey mode across various metrics including determination accuracy, reasoning similarity, and human evaluation scores, although RAG achieved higher cosine similarity scores in Afib cases (0.82 vs. 0.75) and better accuracy in CIED cases (0.70 vs. 0.66), though these differences were not statistically significant due to the small sample size. Our mixed effects analysis revealed no significant effects of temperature or method interactions, indicating stable performance across these variables. Moreover, while no individual evaluation metric, such as BERTScore, BLEURT or cosine similarity, showed a high correlation with human evaluations. However, the ACC-Sim metric, which averages accuracy and cosine similarity, exhibits the highest correlation with human evaluation, with Spearman`s ρ at 0.86 and a P value < 0.001, indicating a significant ordinal correlation between ACC-Sim and human evaluation. This suggests its potential as a surrogate for human evaluation in similar medical scenarios. Conclusion: Our study did not find a significant difference between the RAG and Cold Turkey methods in terms of ChatGPT-4`s clinical decision-making performance in Afib and CIED infection management. The ACC-Sim metric closely aligns with human evaluations in these specific medical contexts and shows promise for integration into a semi-automatic evaluation framework.

What problem does this paper attempt to address?

This paper explores the potential of ChatGPT-4 in clinical decision support for electrophysiology (EP) in cardiology and proposes a semi-automatic evaluation framework. The study enhances the domain expertise of ChatGPT-4 in the EP field by using a retrieval-augmented generation (RAG) method and incorporating the latest evidence base to enhance its underlying database. Additionally, the study compares the performance of ChatGPT-4 under the RAG model and the "cold turkey" model without external knowledge.

The paper analyzes 5 cases of atrial fibrillation and 7 cases of cardiac implantable electronic device (CIED) infection, testing the accuracy of ChatGPT-4 in diagnosis and treatment recommendations through 120 atrial fibrillation experiments and 168 CIED experiments. The study utilizes automatic evaluation metrics such as BERT scores, BLEURT, and cosine similarity, as well as human evaluations for comprehensive assessment. The results indicate that although the RAG model shows slightly better performance in some aspects, there is no significant difference compared to the "cold turkey" model. Furthermore, the study identifies a new metric called ACC-Sim (average of accuracy and cosine similarity) that highly correlates with human evaluations, which could be a useful tool for future semi-automatic evaluation frameworks.

In conclusion, the clinical decision support capability of ChatGPT-4 in atrial fibrillation and CIED infection management is not significantly improved by the RAG model. However, the ACC-Sim metric shows a strong association with human evaluations in specific medical contexts, offering the potential to alleviate the burden of expert assessments. Future research is needed with a larger sample size to further validate these findings.