[Physiologic degeneration and restoration of neurosecretory cells of the nucleus praeopticus in carp and in Cyprinus carpio].

A. Polenov

Abstract:

What problem does this paper attempt to address?

End-To-End Clinical Trial Matching with Large Language Models

Dyke Ferber,Lars Hilgers,Isabella C. Wiest,Marie-Elisabeth Leßmann,Jan Clusmann,Peter Neidlinger,Jiefu Zhu,Georg Wölflein,Jacqueline Lammert,Maximilian Tschochohei,Heiko Böhme,Dirk Jäger,Mihaela Aldea,Daniel Truhn,Christiane Höper,Jakob Nikolas Kather

2024-07-18

Abstract:Matching cancer patients to clinical trials is essential for advancing treatment and patient care. However, the inconsistent format of medical free text documents and complex trial eligibility criteria make this process extremely challenging and time-consuming for physicians. We investigated whether the entire trial matching process - from identifying relevant trials among 105,600 oncology-related clinical trials on <a class="link-external link-http" href="http://clinicaltrials.gov" rel="external noopener nofollow">this http URL</a> to generating criterion-level eligibility matches - could be automated using Large Language Models (LLMs). Using GPT-4o and a set of 51 synthetic Electronic Health Records (EHRs), we demonstrate that our approach identifies relevant candidate trials in 93.3% of cases and achieves a preliminary accuracy of 88.0% when matching patient-level information at the criterion level against a baseline defined by human experts. Utilizing LLM feedback reveals that 39.3% criteria that were initially considered incorrect are either ambiguous or inaccurately annotated, leading to a total model accuracy of 92.7% after refining our human baseline. In summary, we present an end-to-end pipeline for clinical trial matching using LLMs, demonstrating high precision in screening and matching trials to individual patients, even outperforming the performance of qualified medical doctors. Our fully end-to-end pipeline can operate autonomously or with human supervision and is not restricted to oncology, offering a scalable solution for enhancing patient-trial matching in real-world settings.

Computation and Language,Artificial Intelligence
Large Language Model Augmented Clinical Trial Screening

Jacob Beattie,Dylan Owens,Ann Marie Navar,Luiza Giuliani Schmitt,Kimberly Taing,Sarah Neufeld,Daniel Yang,Christian Chukwuma,Ahmed Gul,Dong Soo Lee,Neil Desai,Dominic Moon,Jing Wang,Steve Jiang,Michael Dohopolski

DOI: https://doi.org/10.1101/2024.08.27.24312646

2024-08-28

Abstract:Purpose: Identifying potential participants for clinical trials using traditional manual screening methods is time-consuming and expensive. Structured data in electronic health records (EHR) are often insufficient to capture trial inclusion and exclusion criteria adequately. Large language models (LLMs) offer the potential for improved participant screening by searching text notes in the EHR, but optimal deployment strategies remain unclear. Methods: We evaluated the performance of GPT-3.5 and GPT-4 in screening a cohort of 74 patients (35 eligible, 39 ineligible) using EHR data, including progress notes, pathology reports, and imaging reports, for a phase 2 clinical trial in patients with head and neck cancer. Fourteen trial criteria were evaluated, including stage, histology, prior treatments, underlying conditions, functional status, etc. Manually annotated data served as the ground truth. We tested three prompting approaches (Structured Output (SO), Chain of Thought (CoT), and Self-Discover (SD)). SO and CoT were further tested using expert and LLM guidance (EG and LLM-G, respectively). Prompts were developed and refined using 10 patients from each cohort and then assessed on the remaining 54 patients. Each approach was assessed for accuracy, sensitivity, specificity, and micro F1 score. We explored two eligibility predictions: strict eligibility required meeting all criteria, while proportional eligibility used the proportion of criteria met. Screening time and cost were measured, and a failure analysis identified common misclassification issues. Results: Fifty-four patients were evaluated (25 enrolled, 29 not enrolled). At the criterion level, GPT-3.5 showed a median accuracy of 0.761 (range: 0.554-0.910), with the Structured Out- put + EG approach performing best. GPT-4 demonstrated a median accuracy of 0.838 (range: 0.758-0.886), with the Self-Discover approach achieving the highest Youden Index of 0.729. For strict patient-level eligibility, GPT-3.5's Structured Output + EG approach reached an accuracy of 0.611, while GPT-4's CoT + EG achieved 0.65. Proportional eligibility performed better over- all, with GPT-4's CoT + LLM-G approach having the highest AUC (0.82) and Youden Index (0.60). Screening times ranged from 1.4 to 3 minutes per patient for GPT-3.5 and 7.9 to 12.4 minutes for GPT-4, with costs of $0.02-$0.03 for GPT-3.5 and $0.15-$0.27 for GPT-4. Conclusion: LLMs can be used to identify specific clinical trial criteria but had difficulties identifying patients who met all criteria. Instead, using the proportion of criteria met to flag candidates for manual review maybe a more practical approach. LLM performance varies by prompt, with GPT-4 generally outperforming GPT-3.5, but at higher costs and longer processing times. LLMs should complement, not replace, manual chart reviews for matching patients to clinical trials.
Scaling Clinical Trial Matching Using Large Language Models: A Case Study in Oncology

Cliff Wong,Sheng Zhang,Yu Gu,Christine Moung,Jacob Abel,Naoto Usuyama,Roshanthi Weerasinghe,Brian Piening,Tristan Naumann,Carlo Bifulco,Hoifung Poon

2023-08-19

Abstract:Clinical trial matching is a key process in health delivery and discovery. In practice, it is plagued by overwhelming unstructured data and unscalable manual processing. In this paper, we conduct a systematic study on scaling clinical trial matching using large language models (LLMs), with oncology as the focus area. Our study is grounded in a clinical trial matching system currently in test deployment at a large U.S. health network. Initial findings are promising: out of box, cutting-edge LLMs, such as GPT-4, can already structure elaborate eligibility criteria of clinical trials and extract complex matching logic (e.g., nested AND/OR/NOT). While still far from perfect, LLMs substantially outperform prior strong baselines and may serve as a preliminary solution to help triage patient-trial candidates with humans in the loop. Our study also reveals a few significant growth areas for applying LLMs to end-to-end clinical trial matching, such as context limitation and accuracy, especially in structuring patient information from longitudinal medical records.

Computation and Language,Machine Learning
Learning to match patients to clinical trials using large language models

Maciej Rybinski,Wojciech Kusa,Sarvnaz Karimi,Allan Hanbury

DOI: https://doi.org/10.1016/j.jbi.2024.104734

2024-10-09

Abstract:Objective: This study investigates the use of Large Language Models (LLMs) for matching patients to clinical trials (CTs) within an information retrieval pipeline. Our objective is to enhance the process of patient-trial matching by leveraging the semantic processing capabilities of LLMs, thereby improving the effectiveness of patient recruitment for clinical trials. Methods: We employed a multi-stage retrieval pipeline integrating various methodologies, including BM25 and Transformer-based rankers, along with LLM-based methods. Our primary datasets were the TREC Clinical Trials 2021-23 track collections. We compared LLM-based approaches, focusing on methods that leverage LLMs in query formulation, filtering, relevance ranking, and re-ranking of CTs. Results: Our results indicate that LLM-based systems, particularly those involving re-ranking with a fine-tuned LLM, outperform traditional methods in terms of nDCG and Precision measures. The study demonstrates that fine-tuning LLMs enhances their ability to find eligible trials. Moreover, our LLM-based approach is competitive with state-of-the-art systems in the TREC challenges. The study shows the effectiveness of LLMs in CT matching, highlighting their potential in handling complex semantic analysis and improving patient-trial matching. However, the use of LLMs increases the computational cost and reduces efficiency. We provide a detailed analysis of effectiveness-efficiency trade-offs. Conclusion: This research demonstrates the promising role of LLMs in enhancing the patient-to-clinical trial matching process, offering a significant advancement in the automation of patient recruitment. Future work should explore optimising the balance between computational cost and retrieval effectiveness in practical applications.
Matching Patients to Clinical Trials with Large Language Models

Qiao Jin,Zifeng Wang,Charalampos S. Floudas,Fangyuan Chen,Changlin Gong,Dara Bracken-Clarke,Elisabetta Xue,Yifan Yang,Jimeng Sun,Zhiyong Lu

2024-04-28

Abstract:Clinical trials are often hindered by the challenge of patient recruitment. In this work, we introduce TrialGPT, a first-of-its-kind large language model (LLM) framework to assist patient-to-trial matching. Given a patient note, TrialGPT predicts the patient's eligibility on a criterion-by-criterion basis and then consolidates these predictions to assess the patient's eligibility for the target trial. We evaluate the trial-level prediction performance of TrialGPT on three publicly available cohorts of 184 patients with over 18,000 trial annotations. We also engaged three physicians to label over 1,000 patient-criterion pairs to assess its criterion-level prediction accuracy. Experimental results show that TrialGPT achieves a criterion-level accuracy of 87.3% with faithful explanations, close to the expert performance (88.7%-90.0%). The aggregated TrialGPT scores are highly correlated with human eligibility judgments, and they outperform the best-competing models by 32.6% to 57.2% in ranking and excluding clinical trials. Furthermore, our user study reveals that TrialGPT can significantly reduce the screening time (by 42.6%) in a real-life clinical trial matching task. These results and analyses have demonstrated promising opportunities for clinical trial matching with LLMs such as TrialGPT.

Computation and Language,Artificial Intelligence
Distilling Large Language Models for Matching Patients to Clinical Trials

Mauro Nievas,Aditya Basu,Yanshan Wang,Hrituraj Singh

2023-12-16

Abstract:The recent success of large language models (LLMs) has paved the way for their adoption in the high-stakes domain of healthcare. Specifically, the application of LLMs in patient-trial matching, which involves assessing patient eligibility against clinical trial's nuanced inclusion and exclusion criteria, has shown promise. Recent research has shown that GPT-3.5, a widely recognized LLM developed by OpenAI, can outperform existing methods with minimal 'variable engineering' by simply comparing clinical trial information against patient summaries. However, there are significant challenges associated with using closed-source proprietary LLMs like GPT-3.5 in practical healthcare applications, such as cost, privacy and reproducibility concerns. To address these issues, this study presents the first systematic examination of the efficacy of both proprietary (GPT-3.5, and GPT-4) and open-source LLMs (LLAMA 7B,13B, and 70B) for the task of patient-trial matching. Employing a multifaceted evaluation framework, we conducted extensive automated and human-centric assessments coupled with a detailed error analysis for each model. To enhance the adaptability of open-source LLMs, we have created a specialized synthetic dataset utilizing GPT-4, enabling effective fine-tuning under constrained data conditions. Our findings reveal that open-source LLMs, when fine-tuned on this limited and synthetic dataset, demonstrate performance parity with their proprietary counterparts. This presents a massive opportunity for their deployment in real-world healthcare applications. To foster further research and applications in this field, we release both the annotated evaluation dataset along with the fine-tuned LLM -- Trial-LLAMA -- for public use.

Artificial Intelligence,Information Retrieval
Automated Matching of Patients to Clinical Trials: A Patient-Centric Natural Language Processing Approach for Pediatric Leukemia

Samuel Kaskovich,Kirk D Wyatt,Tomasz Oliwa,Luca Graglia,Brian Furner,Jooho Lee,Anoop Mayampurath,Samuel L Volchenboum

DOI: https://doi.org/10.1200/CCI.23.00009

Abstract:Purpose: Matching patients to clinical trials is cumbersome and costly. Attempts have been made to automate the matching process; however, most have used a trial-centric approach, which focuses on a single trial. In this study, we developed a patient-centric matching tool that matches patient-specific demographic and clinical information with free-text clinical trial inclusion and exclusion criteria extracted using natural language processing to return a list of relevant clinical trials ordered by the patient's likelihood of eligibility. Materials and methods: Records from pediatric leukemia clinical trials were downloaded from ClinicalTrials.gov. Regular expressions were used to discretize and extract individual trial criteria. A multilabel support vector machine (SVM) was trained to classify sentence embeddings of criteria into relevant clinical categories. Labeled criteria were parsed using regular expressions to extract numbers, comparators, and relationships. In the validation phase, a patient-trial match score was generated for each trial and returned in the form of a ranked list for each patient. Results: In total, 5,251 discretized criteria were extracted from 216 protocols. The most frequent criterion was previous chemotherapy/biologics (17%). The multilabel SVM demonstrated a pooled accuracy of 75%. The text processing pipeline was able to automatically extract 68% of eligibility criteria rules, as compared with 80% in a manual version of the tool. Automated matching was accomplished in approximately 4 seconds, as compared with several hours using manual derivation. Conclusion: To our knowledge, this project represents the first open-source attempt to generate a patient-centric clinical trial matching tool. The tool demonstrated acceptable performance when compared with a manual version, and it has potential to save time and money when matching patients to trials.
A Proof-of-Concept Large Language Model Application to Support Clinical Trial Screening in Surgical Oncology

Samantha M Lai,Alysala M Malik,Tejas S Sathe,Caitlin Silvestri,Gulam A Manji,Michael D Kluger

DOI: https://doi.org/10.1101/2024.09.20.24314053

2024-10-04

Abstract:Introduction Clinical trials are the gold standard for advancing the forefront of medical knowledge and rely on consistent patient accrual for success. However, patient screening for clinical trials is complex and resource intensive. There is a need to increase the scalability of trial recruitment while maintaining or improving upon the sensitivity of the current process. We hypothesized that we could use a state-of-the-art large language model, prompt engineering, and publicly available clinical trial data to predict patient eligibility for clinical trials from clinic notes. Here, we present pilot data demonstrating the accuracy of our tool in a cohort of patients being evaluated for pancreas cancer. Methods We used a cohort of patients who were screened for clinical trials at a single institution for this study. We developed an LLM application using LangChain and the GPT-4o model to assist in clinical trial screening. Deidentified patient data from clinical notes and trial eligibility criteria from ClinicalTrials.gov were used as inputs. For each patient and trial, the model determined inclusion or exclusion with selected eligibility criteria as well as the trial overall. Model responses were graded programmatically against a human rater standard. We recorded time elapsed and cost of running each analysis. Results Of the 24 patients in the test set, 19 were eligible for at least one trial. There were 43 eligible patient-trial matches in our data set. Our model correctly predicted 39 out of 42 (90.7%) of these matches. There were 105 individual eligibility criteria evaluated per patient for a total of 2520 binary criteria. GPT-4o agreed with the raters for 2438 out of 2520 (96.7%) binary eligibility criteria. Sensitivity to overall trial eligibility by trial ranged from 87.5% to 100% for 8 out of 9 trials. The application incorrectly screened the single patient in the CA-4948 trial test set, leading to a sensitivity of 0% for this trial. Specificity ranged from 73.3% to 100% over all nine trials. The median cost for screening a patient was 0.67 USD (0.63-0.74). Median time elapsed was 137.66 seconds (130.04-146.04). Median total token usage across three assistants was 112,266.5 tokens (102982.0-122174.2). Conclusion Overall, our model showed high efficiency and accuracy in selecting patients for appropriate clinical trials. Our results showed promise with a small cohort and future studies are needed to assess its accuracy with a larger sample of patients and trials. This model can be applied systematically to widen the surface area covered by the screening process as well as adapted to any set of clinical trials, opening the door for expansion to other surgical and medical disciplines.
PRISM: Patient Records Interpretation for Semantic Clinical Trial Matching using Large Language Models

Shashi Kant Gupta,Aditya Basu,Mauro Nievas,Jerrin Thomas,Nathan Wolfrath,Adhitya Ramamurthi,Bradley Taylor,Anai N. Kothari,Regina Schwind,Therica M. Miller,Sorena Nadaf-Rahrov,Yanshan Wang,Hrituraj Singh

2024-04-27

Abstract:Clinical trial matching is the task of identifying trials for which patients may be potentially eligible. Typically, this task is labor-intensive and requires detailed verification of patient electronic health records (EHRs) against the stringent inclusion and exclusion criteria of clinical trials. This process is manual, time-intensive, and challenging to scale up, resulting in many patients missing out on potential therapeutic options. Recent advancements in Large Language Models (LLMs) have made automating patient-trial matching possible, as shown in multiple concurrent research studies. However, the current approaches are confined to constrained, often synthetic datasets that do not adequately mirror the complexities encountered in real-world medical data. In this study, we present the first, end-to-end large-scale empirical evaluation of clinical trial matching using real-world EHRs. Our study showcases the capability of LLMs to accurately match patients with appropriate clinical trials. We perform experiments with proprietary LLMs, including GPT-4 and GPT-3.5, as well as our custom fine-tuned model called OncoLLM and show that OncoLLM, despite its significantly smaller size, not only outperforms GPT-3.5 but also matches the performance of qualified medical doctors. All experiments were carried out on real-world EHRs that include clinical notes and available clinical trials from a single cancer center in the United States.

Computation and Language,Artificial Intelligence
Zero-Shot Clinical Trial Patient Matching with LLMs

Michael Wornow,Alejandro Lozano,Dev Dash,Jenelle Jindal,Kenneth W. Mahaffey,Nigam H. Shah

2024-04-10

Abstract:Matching patients to clinical trials is a key unsolved challenge in bringing new drugs to market. Today, identifying patients who meet a trial's eligibility criteria is highly manual, taking up to 1 hour per patient. Automated screening is challenging, however, as it requires understanding unstructured clinical text. Large language models (LLMs) offer a promising solution. In this work, we explore their application to trial matching. First, we design an LLM-based system which, given a patient's medical history as unstructured clinical text, evaluates whether that patient meets a set of inclusion criteria (also specified as free text). Our zero-shot system achieves state-of-the-art scores on the n2c2 2018 cohort selection benchmark. Second, we improve the data and cost efficiency of our method by identifying a prompting strategy which matches patients an order of magnitude faster and more cheaply than the status quo, and develop a two-stage retrieval pipeline that reduces the number of tokens processed by up to a third while retaining high performance. Third, we evaluate the interpretability of our system by having clinicians evaluate the natural language justifications generated by the LLM for each eligibility decision, and show that it can output coherent explanations for 97% of its correct decisions and 75% of its incorrect ones. Our results establish the feasibility of using LLMs to accelerate clinical trial operations.

Artificial Intelligence
Performance of a trained large language model to provide clinical trial recommendation in a head and neck cancer population.

Tony Hung,Gilad Kuperman,Eric Jeffrey Sherman,Alan Loh Ho,Winston Wong,Anuja Kriplani,Lara Dunn,James Vincent Fetten,Loren S. Michel,Shrujal S. Baxi,Chunhua Weng,David G. Pfister,Jun J. Mao

DOI: https://doi.org/10.1200/jco.2024.42.16_suppl.11081

IF: 45.3

2024-06-01

Journal of Clinical Oncology

Abstract:11081 Background: Chatbots based on large language model (LLM) have demonstrated ability to answer oncology exam questions; however, leveraging LLM in medical-decision support have not yet demonstrated suitable performance in oncology practice. We evaluated the performance of a trained a LLM, GPT-4, to recommend appropriate clinical trials for a head & neck (HN) cancer population. Methods: In 2022, we developed an artificial intelligence powered clinical trial management mobile app, LookUpTrials, and demonstrated promising user engagement among oncologists. Using LookUpTrials database, we applied direct preference optimization to train GPT-4 as an in-app assistant to LookUpTrials. From Nov 7 to Dec 19, 2023, we collected consecutive, new patient cases and their respective clinical trial recommendations from oncologists in the HN medical oncology service at Memorial Sloan Kettering Cancer Center. Cases were categorized by diagnosis, cancer stage, treatment setting, and physician recommendation on clinical trials. Trained GPT-4 is prompted using a semi-structured template: “Given patient with a , , , what are possible clinical trials?” Physician recommendations were compared with trained GPT-4 responses. We analyzed the performance of GPT-4 based on its response precision (positive predictive value), recall (sensitivity), and F1 score (harmonic mean of precision and recall). Results: We analyzed 178 patient cases, mean age 65.6 (SD 13.9), primarily male (75%) with local/locally advanced (68%) HN (61%), thyroid (16%), skin (9%), or salivary (8%) cancers. Majority were treated in the definitive setting with combined modality therapy (42%) and modest proportion were treated under clinical trials (10%). Overall, trained GPT-4 achieved a moderate performance matching physician clinical trial recommendations with 63% precision and 100% recall (F1 score 0.77), narrowing a total list of 56 HN clinical trials to a range of 0-4 relevant trials per patient case (mean 1, SD 1.2). Comparatively, performance of our trained GPT-4 exceeded historic performance of untrained LLMs to provide oncology treatment recommendation by 4-20 folds (F1 score 0.04 - 0.19). Conclusions: This proof-of-concept study demonstrated that trained LLM can achieve moderate performance in matching physician clinical trial recommendation in HN oncology. Our results suggest the potential of embedding trained LLM into oncology workflow to aid clinical trial search and accelerate clinical trial accrual. Future research is needed to optimize precision of trained LLM and to assess whether trained LLM may be a scalable solution to enhance the diversity and equity of clinical trial participation.

oncology
Transforming clinical trials: the emerging roles of large language models

Jong-Lyul Ghim,Sangzin Ahn

DOI: https://doi.org/10.12793/tcp.2023.31.e16

2023-01-01

Translational and Clinical Pharmacology

Abstract:Clinical trials are essential for medical research, but they often face challenges in matching patients to trials and planning. Large language models (LLMs) offer a promising solution, signaling a transformative shift in the field of clinical trials. This review explores the multifaceted applications of LLMs within clinical trials, focusing on five main areas expected to be implemented in the near future: enhancing patient-trial matching, streamlining clinical trial planning, analyzing free text narratives for coding and classification, assisting in technical writing tasks, and providing cognizant consent via LLM-powered chatbots. While the application of LLMs is promising, it poses challenges such as accuracy validation and legal concerns. The convergence of LLMs with clinical trials has the potential to revolutionize the efficiency of clinical trials, paving the way for innovative methodologies and enhancing patient engagement. However, this development requires careful consideration and investment to overcome potential hurdles.
Retrieval-augmented large language models for clinical trial screening.

Ryan Tan,Si Xian Ho,Shiyun Vivianna Fequira Oo,Shi Ling Chua,Ma Wai Wai Zaw,Daniel Shao-Weng Tan

DOI: https://doi.org/10.1200/jco.2024.42.16_suppl.e13611

IF: 45.3

2024-05-31

Journal of Clinical Oncology

Abstract:e13611 Background: Clinical trial screening is currently manual and laborious. We tested several large language models enhanced with retrieval-augmented generation (RAG-LLM) to assess their performance on this task. Methods: We extracted eligibility criteria of 184 oncology trials with FDA approval notifications between 8 January 2020 and 18 January 2024, as well as information on cancer staging and performance status scoring for the RAG-LLM vector database. A medical oncologist and 2 senior clinical trial coordinators developed a test set of 975 synthetic patient profiles which included primary site, stage, prior therapy, tumor mutations and one additional clinical feature. Each profile was paired with one of the 184 trials and annotated for ground-truth eligibility and the reason for it. This process was repeated for another validation set of 240 longer and more challenging profiles paired with 8 ongoing trials. We developed RAG-LLMs with 4 leading LLMs (Zephyr-7B, Med42, GPT 3.5, GPT4) and evaluated their accuracy in determining trial eligibility as well as retrieval-augmented generation assessment (RAGAs) metrics. A response was deemed accurate only if it correctly assigned both trial eligibility and the reason for assignment. Results: GPT4 performed best and achieved an accuracy of 95.18% and 80.00% on the test and validation set respectively with a mean inference time of 10.95 seconds. It also demonstrated the highest answer relevancy, context relevancy and faithfulness (Table). Conclusions: Our results demonstrate potential for RAG-LLMs to assist with trial screening at scale. Further evaluation in real-world cohorts utilizing electronic records and full protocol data with tracking of impact on trial enrolment can be explored within secure firewalls. [Table: see text]

oncology
PRISM: Patient Records Interpretation for Semantic clinical trial Matching system using large language models

Shashi Gupta,Aditya Basu,Mauro Nievas,Jerrin Thomas,Nathan Wolfrath,Adhitya Ramamurthi,Bradley Taylor,Anai N. Kothari,Regina Schwind,Therica M. Miller,Sorena Nadaf-Rahrov,Yanshan Wang,Hrituraj Singh

DOI: https://doi.org/10.1038/s41746-024-01274-7

IF: 15.2

2024-10-30

npj Digital Medicine

Abstract:Clinical trial matching is the task of identifying trials for which patients may be eligible. Typically, this task is labor-intensive and requires detailed verification of patient electronic health records (EHRs) against the stringent inclusion and exclusion criteria of clinical trials. This process also results in many patients missing out on potential therapeutic options. Recent advancements in Large Language Models (LLMs) have made automating patient-trial matching possible, as shown in multiple concurrent research studies. However, the current approaches are confined to constrained, often synthetic, datasets that do not adequately mirror the complexities encountered in real-world medical data. In this study, we present an end-to-end large-scale empirical evaluation of a clinical trial matching system and validate it using real-world EHRs. We perform comprehensive experiments with proprietary LLMs and our custom fine-tuned model called OncoLLM and show that OncoLLM outperforms GPT-3.5 and matches the performance of qualified medical doctors for clinical trial matching.

health care sciences & services,medical informatics
Enhancing Biomarker-Based Oncology Trial Matching Using Large Language Models

Nour Al Khoury,Maqsood Shaik,Ricardo Wurmus,Altuna Akalin

DOI: https://doi.org/10.1101/2024.09.13.612922

2024-09-19

Abstract:Clinical trials are an essential component of drug development for new cancer treatments, yet the information required to determine a patient’s eligibility for enrollment is scattered in large amounts of unstructured text. Genomic biomarkers are especially important in precision medicine and targeted therapies, making them essential for matching patients to appropriate trials. Large language models (LLMs) offer a promising solution for extracting this information from clinical trial data, aiding both physicians and patients in identifying suitable matches. In this study, we explore various LLM strategies for extracting genetic biomarkers from oncology trials to improve patient enrollment rates. Our results show that open-source language models, when applied out-of-the-box, effectively capture complex logical expressions and structure genomic biomarkers in disjunctive normal form, outperforming closed-source models such as GPT-4 and GPT-3.5-Turbo. Furthermore, fine-tuning these open-source models with additional data significantly enhances their performance.

Bioinformatics
Epilogue. Establishment of the Center for Chronic Disease Prevention and Health Promotion.

W. R. Taylor,A. Plott,S. Cheek,F. Martín,R. Rothenberg

Acta Cytologica

Abstract:The establishment of the Center for Chronic Disease Prevention and Health Promotion (CCDPHP) at the Centers for Disease Control (CDC) following the Conferences on the State of the Art in Quality Control Measures for Diagnostic Cytology Laboratories is briefly discussed. The CCDPHP is expected to play a major role in the CDC's cancer control program, including participation in establishing effective screening programs and assuring the quality of such methodologies as the Papanicolaou test and mammography.
[Usefulness of the Michigan Alcoholism Screening Test (MAST) in Poland].

Z. Falicki,J. Karczewski,L. Wandzel,W. Chrzanowski

Psychiatria Polska

Abstract:
Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study

Eddie Guo,Mehul Gupta,Jiawen Deng,Ye-Jean Park,Michael Paget,Christopher Naugler

DOI: https://doi.org/10.2196/48996

2024-01-12

Abstract:Background: The systematic review of clinical research papers is a labor-intensive and time-consuming process that often involves the screening of thousands of titles and abstracts. The accuracy and efficiency of this process are critical for the quality of the review and subsequent health care decisions. Traditional methods rely heavily on human reviewers, often requiring a significant investment of time and resources. Objective: This study aims to assess the performance of the OpenAI generative pretrained transformer (GPT) and GPT-4 application programming interfaces (APIs) in accurately and efficiently identifying relevant titles and abstracts from real-world clinical review data sets and comparing their performance against ground truth labeling by 2 independent human reviewers. Methods: We introduce a novel workflow using the Chat GPT and GPT-4 APIs for screening titles and abstracts in clinical reviews. A Python script was created to make calls to the API with the screening criteria in natural language and a corpus of title and abstract data sets filtered by a minimum of 2 human reviewers. We compared the performance of our model against human-reviewed papers across 6 review papers, screening over 24,000 titles and abstracts. Results: Our results show an accuracy of 0.91, a macro F1-score of 0.60, a sensitivity of excluded papers of 0.91, and a sensitivity of included papers of 0.76. The interrater variability between 2 independent human screeners was κ=0.46, and the prevalence and bias-adjusted κ between our proposed methods and the consensus-based human decisions was κ=0.96. On a randomly selected subset of papers, the GPT models demonstrated the ability to provide reasoning for their decisions and corrected their initial decisions upon being asked to explain their reasoning for incorrect classifications. Conclusions: Large language models have the potential to streamline the clinical review process, save valuable time and effort for researchers, and contribute to the overall quality of clinical reviews. By prioritizing the workflow and acting as an aid rather than a replacement for researchers and reviewers, models such as GPT-4 can enhance efficiency and lead to more accurate and reliable conclusions in medical research.
Effect of a novel artificial intelligence (AI) –enabled multi-trial matching system on patient matching using real-world data.

Selin Kurnaz,Arturo Loaiza-Bonilla,Darla Carvallo Castañeda,Oz Huner,Precision Cancer Consortium

DOI: https://doi.org/10.1200/jco.2024.42.16_suppl.e13501

IF: 45.3

2024-05-31

Journal of Clinical Oncology

Abstract:e13501 Background: Currently clinical research teams at sites are manually matching patients to clinical trials, where each trial in our experience takes at least 25 minutes using highly trained dedicated resources. This limits cancer patients’ clinical trial options and enrollment rates. Moreover, the lack of NGS testing results may also negatively impact matching to clinical trials in targeted therapies. PCC, a consortium of 7 of the Top 20 Pharmaceutical companies, collaborated with Massive Bio to demonstrate how cancer patients’ matching rates can be increased if patients are prescreened to multiple trials simultaneously (i.e. multi-trial matching) instead of a single trial and NGS testing results are integrated into the matching process. Methods: Massive Bio has developed a first-in-class AI-enabled matching system to extract 180 structured clinical data from medical records using computer vision and natural language processing and a recommendation system for matching patients to parameterized inclusion/exclusion (I/E) criteria from over 14,000 actively recruiting interventional clinical trials.Our analysis included a sample of 5,707 cancer patients from the Massive Bio Synergy AI real-world database, corresponding to tumor types in 23 selected trials. We developed a decision-tree algorithm for retrospective matching analysis using modeled I/E criteria and compared these results to the theoretical matching rate (TMR) for each trial based on tumor type and biomarker prevalence, extent of disease at diagnosis and prior treatment history. Results: 690 patients matched to at least one trial were matched to 1.82 times more trials in multi-trial matching; 2.10 times more potential matches were obtained with hypothetical 100% NGS testing; and 99.9% matching time was reduced with AI enabled system; key findings are summarized (Table). Conclusions: This study confirms the underutilization of NGS technology and substantial benefits of AI and NGS in cancer trial matching. Their use resulted in a nearly two-fold increase in potential patient eligibility for trials, and an unprecedented reduction in manual effort, underscoring a significant advancement in the efficiency of the clinical trial matching process. A prospective country-wide analysis is under consideration. [Table: see text]

oncology

[Physiologic degeneration and restoration of neurosecretory cells of the nucleus praeopticus in carp and in Cyprinus carpio].

End-To-End Clinical Trial Matching with Large Language Models

Large Language Model Augmented Clinical Trial Screening

Scaling Clinical Trial Matching Using Large Language Models: A Case Study in Oncology

Learning to match patients to clinical trials using large language models

Matching Patients to Clinical Trials with Large Language Models

Distilling Large Language Models for Matching Patients to Clinical Trials

Automated Matching of Patients to Clinical Trials: A Patient-Centric Natural Language Processing Approach for Pediatric Leukemia

A Proof-of-Concept Large Language Model Application to Support Clinical Trial Screening in Surgical Oncology

PRISM: Patient Records Interpretation for Semantic Clinical Trial Matching using Large Language Models

Zero-Shot Clinical Trial Patient Matching with LLMs

Performance of a trained large language model to provide clinical trial recommendation in a head and neck cancer population.

Transforming clinical trials: the emerging roles of large language models

Retrieval-augmented large language models for clinical trial screening.

PRISM: Patient Records Interpretation for Semantic clinical trial Matching system using large language models

Enhancing Biomarker-Based Oncology Trial Matching Using Large Language Models

Epilogue. Establishment of the Center for Chronic Disease Prevention and Health Promotion.

[Usefulness of the Michigan Alcoholism Screening Test (MAST) in Poland].

Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study

Effect of a novel artificial intelligence (AI) –enabled multi-trial matching system on patient matching using real-world data.