Enhancing Large Language Models for Clinical Decision Support by Incorporating Clinical Practice Guidelines

David Oniani,Xizhi Wu,Shyam Visweswaran,Sumit Kapoor,Shravan Kooragayalu,Katelyn Polanska,Yanshan Wang
2024-01-24
Abstract:Background Large Language Models (LLMs), enhanced with Clinical Practice Guidelines (CPGs), can significantly improve Clinical Decision Support (CDS). However, methods for incorporating CPGs into LLMs are not well studied. Methods We develop three distinct methods for incorporating CPGs into LLMs: Binary Decision Tree (BDT), Program-Aided Graph Construction (PAGC), and Chain-of-Thought-Few-Shot Prompting (CoT-FSP). To evaluate the effectiveness of the proposed methods, we create a set of synthetic patient descriptions and conduct both automatic and human evaluation of the responses generated by four LLMs: GPT-4, GPT-3.5 Turbo, LLaMA, and PaLM 2. Zero-Shot Prompting (ZSP) was used as the baseline method. We focus on CDS for COVID-19 outpatient treatment as the case study. Results All four LLMs exhibit improved performance when enhanced with CPGs compared to the baseline ZSP. BDT outperformed both CoT-FSP and PAGC in automatic evaluation. All of the proposed methods demonstrated high performance in human evaluation. Conclusion LLMs enhanced with CPGs demonstrate superior performance, as compared to plain LLMs with ZSP, in providing accurate recommendations for COVID-19 outpatient treatment, which also highlights the potential for broader applications beyond the case study.
Computation and Language,Artificial Intelligence
What problem does this paper attempt to address?
The problem that this paper attempts to solve is how to enhance the performance of large language models (LLMs) in clinical decision support (CDS) by combining clinical practice guidelines (CPGs). Specifically, the paper focuses on how to effectively integrate CPGs into LLMs to improve their accuracy in providing outpatient treatment recommendations for COVID - 19. ### Background Although LLMs have demonstrated strong capabilities in multiple fields, in the application in the healthcare field, especially in clinical decision support, how to effectively integrate clinical practice guidelines is a problem that has not been fully studied. Most of the existing methods rely on zero - shot prompting (ZSP), but the effectiveness of this method is limited. ### Methods To improve this situation, the paper proposes three different methods to integrate CPGs into LLMs: 1. **Binary Decision Tree (BDT)**: By constructing a binary tree structure, use LLM for recursive querying, gradually narrow the decision - making scope, and finally reach specific treatment recommendations. 2. **Program - Aided Graph Construction (PAGC)**: Include a network graph (defined using the NetworkX library) in the prompt, and select the appropriate path through LLM to finally determine the treatment plan. 3. **Chain - of - Thought - Few - Shot Prompting (CoT - FSP)**: By providing several examples and a step - by - step reasoning process, guide LLM to generate more accurate treatment recommendations. ### Evaluation To evaluate the effectiveness of these methods, the researchers created a set of synthetic patient descriptions and carried out automatic and manual evaluations using four different LLMs (GPT - 4, GPT - 3.5 Turbo, LLaMA, PaLM 2). The baseline method is zero - shot prompting (ZSP). ### Results The experimental results show that all four LLMs exhibit better performance than the baseline method after combining with CPGs. Among them, BDT performs best in the automatic evaluation, and all the proposed methods perform excellently in the manual evaluation. ### Conclusion By combining CPGs, LLMs show significant advantages in providing accurate outpatient treatment recommendations for COVID - 19. This not only improves the accuracy of clinical decision support but also provides potential possibilities for applications in other fields. ### Future Directions The paper also proposes future research directions, including further optimizing the three proposed methods, exploring completely different methods for LLM - enhanced CDS, and developing transparent and privacy - protected LLM systems. In addition, the researchers also emphasize the need to pay special attention to data privacy and security when using LLMs in the medical field.