ChatGPT-CARE: a Superior Decision Support Tool Enhancing ChatGPT with Clinical Practice Guidelines

Yanshan Wang,Shyam Visweswaran,Sumit Kapoor,Shravan Kooragayalu,Xizhi Wu

DOI: https://doi.org/10.1101/2023.08.09.23293890

2024-03-06

Abstract:ChatGPT has gained remarkable traction since its inception in November 2022. However, it faces limitations in generating inaccurate responses, ignoring existing guidelines, and lacking reasoning when applied in clinical settings. This study introduces ChatGPT-CARE, a tool that integrates clinical practice guidelines with ChatGPT, focusing on COVID-19 outpatient treatment decisions. By employing in-context learning and chain-of-thought prompting techniques, ChatGPT-CARE enhances original ChatGPT’s clinical decision support and reasoning capabilities. We created three categories of various descriptions of patients seeking COVID-19 treatment to evaluate the proposed tool, and asked two physicians specialized in pulmonary disease and critical care to assess the responses for accuracy, hallucination, and clarity. The results indicate that ChatGPT-CARE offers increased accuracy and clarity, with moderate hallucination, compared to the original ChatGPT. The proposal ChatGPT-CARE could be a viable AI-driven clinical decision support tool superior to ChatGPT, with potential applications beyond COVID-19 treatment decision support.

Health Informatics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to enhance the accuracy and reasoning ability of ChatGPT in clinical decision - making support through the integration of clinical practice guidelines, especially in COVID - 19 outpatient treatment decisions. Specifically, the paper addresses the following challenges:

1. **Generating inaccurate responses**: The original ChatGPT sometimes ignores important context information, resulting in inaccurate responses.
2. **Ignoring existing guidelines**: ChatGPT may not follow existing clinical practice guidelines when providing medical advice.
3. **Lacking reasoning ability**: When applied in a clinical setting, ChatGPT has limited reasoning ability and it is difficult to provide a transparent decision - making process.

To address these challenges, the paper introduces **ChatGPT - CARE**, a tool for enhancing ChatGPT by integrating clinical practice guidelines. This tool is mainly implemented through the following techniques:

- **In - context Learning**: Through predefined prompts, the model can be fine - tuned on specific tasks without additional training.
- **Chain - of - Thought Prompting**: Improve the model's reasoning ability through a series of intermediate reasoning steps, simulating the human thinking process.

The paper evaluates the performance of ChatGPT - CARE through three categories of patient descriptions with different levels of complexity (simple, medium, difficult), and two professional doctors evaluate the generated responses, mainly from three aspects: accuracy, hallucination, and clarity.

Finally, the research results show that ChatGPT - CARE exhibits higher accuracy and clarity in all patient description categories, especially in reasoning and processing complex information. This makes ChatGPT - CARE a potential clinical decision - making support tool that is superior to the original ChatGPT.