Evaluating GPT-4 as a Clinical Decision Support Tool in Ischemic Stroke Management

Amit Haim,Mark Katson,Michal Cohen-Shelly,Shlomi Peretz,Dvir Aran,Shahar Shelly

DOI: https://doi.org/10.1101/2024.01.18.24301409

2024-01-25

Abstract:Cerebrovascular diseases are the second most common cause of death worldwide and one of the major causes of disability burden. Advancements in artificial intelligence (AI) have the potential to revolutionize healthcare delivery, particularly in critical decision-making scenarios such as ischemic stroke management. This study evaluates the effectiveness of GPT-4 in providing clinical decision support for emergency room neurologists by comparing its recommendations with expert opinions and real-world treatment outcomes. A cohort of 100 consecutive patients with acute stroke symptoms was retrospectively reviewed. The data used for decision making included patients’ history, clinical evaluation, imaging studies results, and other relevant details. Each case was independently presented to GPT-4, which provided a scaled recommendation (1-7) regarding the appropriateness of treatment, the use of tissue plasminogen activator (tPA), and the need for endovascular thrombectomy (EVT). Additionally, GPT-4 estimated the 90-day mortality probability for each patient and elucidated its reasoning for each recommendation. The recommendations were then compared with those of a stroke specialist and actual treatment decision. The agreement of GPT-4’s recommendations with the expert opinion yielded an Area Under the Curve (AUC) of 0.85 [95% CI: 0.77-0.93], and with real-world treatment decisions, an AUC of 0.80 [0.69-0.91]. In terms of mortality prediction, out of 13 patients who died within 90 days, GPT-4 accurately identified 10 within its top 25 high-risk predictions (AUC = 0.89 [95% CI: 0.8077-0.9739]; HR: 6.98 [95% CI: 2.88-16.9]), surpassing supervised machine-learning models. This study demonstrates the potential of GPT-4 as a viable clinical decision support tool in the management of ischemic stroke. Its ability to provide explainable recommendations without requiring structured data input aligns well with the routine workflows of treating physicians. Future studies should focus on prospective validations and exploring the integration of such AI tools into clinical practice.

Neurology

What problem does this paper attempt to address?

This paper evaluated the clinical decision support role of GPT-4 in the management of acute ischemic stroke. The study analyzed the effectiveness of GPT-4 in providing treatment appropriateness, tissue plasminogen activator (tPA) use, and endovascular thrombectomy (EVT) demand by comparing its recommendations with expert opinions and actual treatment outcomes. GPT-4 also predicted the 90-day mortality rate.

The study retrospectively analyzed the cases of 100 stroke symptomatic patients and found that the area under the curve (AUC) for the consistency between GPT-4 recommendations and expert opinions was 0.85, and the AUC for consistency with actual treatment decisions was 0.80. In terms of predicting the 90-day mortality rate, GPT-4 accurately identified 10 out of 13 deceased patients, outperforming supervised machine learning models.

GPT-4 is able to provide interpretable recommendations based on unstructured data such as medical history, clinical assessments, and imaging results, adapting to physicians' daily workflows. However, the study also pointed out limitations of GPT-4 in handling complex cases and ethical issues, emphasizing the need for prospective validation and exploration of integrating AI tools into clinical practice in the future.

Overall, this paper discusses the potential of GPT-4 as a clinical decision support tool in stroke management, indicating its potential to improve the accuracy and efficiency of medical decisions. However, further research is needed to validate its effectiveness and address potential issues.