Large Language Model Augmented Clinical Trial Screening

Jacob Beattie,Dylan Owens,Ann Marie Navar,Luiza Giuliani Schmitt,Kimberly Taing,Sarah Neufeld,Daniel Yang,Christian Chukwuma,Ahmed Gul,Dong Soo Lee,Neil Desai,Dominic Moon,Jing Wang,Steve Jiang,Michael Dohopolski

DOI: https://doi.org/10.1101/2024.08.27.24312646

2024-08-28

Abstract:Purpose: Identifying potential participants for clinical trials using traditional manual screening methods is time-consuming and expensive. Structured data in electronic health records (EHR) are often insufficient to capture trial inclusion and exclusion criteria adequately. Large language models (LLMs) offer the potential for improved participant screening by searching text notes in the EHR, but optimal deployment strategies remain unclear. Methods: We evaluated the performance of GPT-3.5 and GPT-4 in screening a cohort of 74 patients (35 eligible, 39 ineligible) using EHR data, including progress notes, pathology reports, and imaging reports, for a phase 2 clinical trial in patients with head and neck cancer. Fourteen trial criteria were evaluated, including stage, histology, prior treatments, underlying conditions, functional status, etc. Manually annotated data served as the ground truth. We tested three prompting approaches (Structured Output (SO), Chain of Thought (CoT), and Self-Discover (SD)). SO and CoT were further tested using expert and LLM guidance (EG and LLM-G, respectively). Prompts were developed and refined using 10 patients from each cohort and then assessed on the remaining 54 patients. Each approach was assessed for accuracy, sensitivity, specificity, and micro F1 score. We explored two eligibility predictions: strict eligibility required meeting all criteria, while proportional eligibility used the proportion of criteria met. Screening time and cost were measured, and a failure analysis identified common misclassification issues. Results: Fifty-four patients were evaluated (25 enrolled, 29 not enrolled). At the criterion level, GPT-3.5 showed a median accuracy of 0.761 (range: 0.554-0.910), with the Structured Out- put + EG approach performing best. GPT-4 demonstrated a median accuracy of 0.838 (range: 0.758-0.886), with the Self-Discover approach achieving the highest Youden Index of 0.729. For strict patient-level eligibility, GPT-3.5's Structured Output + EG approach reached an accuracy of 0.611, while GPT-4's CoT + EG achieved 0.65. Proportional eligibility performed better over- all, with GPT-4's CoT + LLM-G approach having the highest AUC (0.82) and Youden Index (0.60). Screening times ranged from 1.4 to 3 minutes per patient for GPT-3.5 and 7.9 to 12.4 minutes for GPT-4, with costs of $0.02-$0.03 for GPT-3.5 and $0.15-$0.27 for GPT-4. Conclusion: LLMs can be used to identify specific clinical trial criteria but had difficulties identifying patients who met all criteria. Instead, using the proportion of criteria met to flag candidates for manual review maybe a more practical approach. LLM performance varies by prompt, with GPT-4 generally outperforming GPT-3.5, but at higher costs and longer processing times. LLMs should complement, not replace, manual chart reviews for matching patients to clinical trials.

What problem does this paper attempt to address?

This paper attempts to solve the problem of low - efficiency patient screening in clinical trials. Traditional manual screening methods are time - consuming and costly, and the structured data in electronic health records (EHR) are often insufficient to fully capture the inclusion and exclusion criteria of the trial. Large language models (LLMs) offer the potential to improve patient screening by searching text notes in EHR, but the optimal deployment strategy remains unclear. Therefore, this study aims to evaluate the performance of using GPT - 3.5 and GPT - 4 for patient screening under different prompting methods to determine the best LLM application strategy and improve the screening efficiency and accuracy of patients in clinical trials.

Specifically, this study focuses on the following aspects:
1. **Evaluating the performance of LLM in patient screening**: By comparing different prompting methods (such as structured output, chain - of - thought, and self - discovery), as well as the effects of expert - guided and LLM - guided, to evaluate the performance of GPT - 3.5 and GPT - 4 in screening patients for head - and - neck cancer phase 2 clinical trials.
2. **Exploring strict patient eligibility and proportional eligibility**: Studied the patient eligibility that strictly requires all criteria to be met, and the method of determining patient eligibility based on the proportion of criteria met, to find out which method is more effective.
3. **Analyzing screening time and cost**: Measured the screening time and cost of each method to evaluate its feasibility and economy in practical applications.
4. **Failure analysis**: Analyzed misclassified cases to identify common misclassification problems, so as to further optimize the application strategy of LLM.

Through these research objectives, the paper aims to provide a more efficient and accurate method for patient screening in clinical trials, thereby increasing the participation rate and research quality of clinical trials.