Abstract:Purpose: Identifying potential participants for clinical trials using traditional manual screening methods is time-consuming and expensive. Structured data in electronic health records (EHR) are often insufficient to capture trial inclusion and exclusion criteria adequately. Large language models (LLMs) offer the potential for improved participant screening by searching text notes in the EHR, but optimal deployment strategies remain unclear. Methods: We evaluated the performance of GPT-3.5 and GPT-4 in screening a cohort of 74 patients (35 eligible, 39 ineligible) using EHR data, including progress notes, pathology reports, and imaging reports, for a phase 2 clinical trial in patients with head and neck cancer. Fourteen trial criteria were evaluated, including stage, histology, prior treatments, underlying conditions, functional status, etc. Manually annotated data served as the ground truth. We tested three prompting approaches (Structured Output (SO), Chain of Thought (CoT), and Self-Discover (SD)). SO and CoT were further tested using expert and LLM guidance (EG and LLM-G, respectively). Prompts were developed and refined using 10 patients from each cohort and then assessed on the remaining 54 patients. Each approach was assessed for accuracy, sensitivity, specificity, and micro F1 score. We explored two eligibility predictions: strict eligibility required meeting all criteria, while proportional eligibility used the proportion of criteria met. Screening time and cost were measured, and a failure analysis identified common misclassification issues. Results: Fifty-four patients were evaluated (25 enrolled, 29 not enrolled). At the criterion level, GPT-3.5 showed a median accuracy of 0.761 (range: 0.554-0.910), with the Structured Out- put + EG approach performing best. GPT-4 demonstrated a median accuracy of 0.838 (range: 0.758-0.886), with the Self-Discover approach achieving the highest Youden Index of 0.729. For strict patient-level eligibility, GPT-3.5's Structured Output + EG approach reached an accuracy of 0.611, while GPT-4's CoT + EG achieved 0.65. Proportional eligibility performed better over- all, with GPT-4's CoT + LLM-G approach having the highest AUC (0.82) and Youden Index (0.60). Screening times ranged from 1.4 to 3 minutes per patient for GPT-3.5 and 7.9 to 12.4 minutes for GPT-4, with costs of $0.02-$0.03 for GPT-3.5 and $0.15-$0.27 for GPT-4. Conclusion: LLMs can be used to identify specific clinical trial criteria but had difficulties identifying patients who met all criteria. Instead, using the proportion of criteria met to flag candidates for manual review maybe a more practical approach. LLM performance varies by prompt, with GPT-4 generally outperforming GPT-3.5, but at higher costs and longer processing times. LLMs should complement, not replace, manual chart reviews for matching patients to clinical trials.

Large language models for abstract screening in systematic- and scoping reviews: A diagnostic test accuracy study

High-performance automated abstract screening with large language model ensembles

Evaluating the effectiveness of large language models in abstract screening: a comparative analysis

Can large language models replace humans in the systematic review process? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages

Large Language Model Augmented Clinical Trial Screening

Can large language models replace humans in systematic reviews? Evaluating GPT‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languages

A Novel Question-Answering Framework for Automated Abstract Screening Using Large Language Models

Exploring the use of a Large Language Model for data extraction in systematic reviews: a rapid feasibility study

Performance of a Large Language Model in Screening Citations

The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic review

The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews

A question-answering framework for automated abstract screening using large language models

Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain

Human-Comparable Sensitivity of Large Language Models in Identifying Eligible Studies Through Title and Abstract Screening: 3-Layer Strategy Using GPT-3.5 and GPT-4 for Systematic Reviews

Large language models for conducting systematic reviews: on the rise, but not yet ready for use - a scoping review

Leveraging large language models for systematic reviewing: A case study using HIV medication adherence research

Collaborative Large Language Models for Automated Data Extraction in Living Systematic Reviews

Zero-shot Generative Large Language Models for Systematic Review Screening Automation

Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study

Assessing the performance of large language models in literature screening for pharmacovigilance: a comparative study

Transformative potential of Large Language Models in data mining on Electronic Health Records.