Abstract:e13609 Background: Precision oncology revolutionized cancer treatment by identifying molecular biomarkers to guide personalized care. The ever-growing body of medical literature presents a challenge for oncologists researching targeted therapies. While recent studies investigated large language models (LLMs) to streamline this process, LLM reliance on general rather than medical knowledge limits clinical relevance and trustworthiness. To address these limitations, we developed a retrieval augmented generation (RAG) system that integrates PubMed clinical studies, trial databases and oncological guidelines with LLMs to support targeted treatment recommendations. The Molecular Tumor Board (MTB) at the Center of Personalized Medicine (ZPM TUM ) guided and evaluated treatment options proposed by the LLM to assess their applicability for clinical decision support. Methods: We used 10 publicly accessible fictional patient cases with 7 tumor types and 59 distinct molecular alterations. Our LLM system MEREDITH (Medical Evidence Retrieval and Data Integration for Tailored Healthcare) consists of Google's Gemini Pro, enhanced with RAG and Chain-of-Thought (CoT) prompting. To establish a benchmark, clinical experts at ZPM TUM manually annotated the cases. Informed by MTB expert feedback, we iteratively improved our LLM system from a draft system relying on PubMed-indexed data to an enhanced system, which replicated expert annotation processes by incorporating oncology guidelines, drug availability and trial databases (ClinicalTrials.gov, QuickQueck.de). ZPM TUM assessed credibility and clinical relevance of manually annotated and LLM-generated recommendations. Patient-level data on (likely) pathogenic molecular alterations and recommended treatment options were summarized using median and interquartile range (IQR). Semantic similarity between LLM and clinician responses was assessed using cosine similarity of text vector embeddings; paired t-test evaluated significance. Results: The median of (likely) pathogenic molecular alterations per patient was 2.5 (IQR: 2-3). ZPM TUM identified a median of 2 treatment options per patient (IQR: 1-3), while the enhanced LLM identified a median of 4 (IQR: 3-5). MEREDITH proposed multiple relevant treatment suggestions, including therapies based on preclinical studies, and molecular interactions, for further assessment by the MTB. ZPM TUM prioritized the most suitable clinical option. The mean semantic textual similarity of LLM responses increased significantly from 0.69 in the draft system to 0.76 in the enhanced system (p <0.001). Thus, feedback from ZPM TUM enhanced the model's ability to align its responses with clinician thought processes. Conclusions: Leveraging expert thought processes to instruct LLMs holds promise as a novel decision support tool for precision oncology.

Prediction of tumor board procedural recommendations using large language models

A Large Language Model Pipeline for Breast Cancer Oncology

Evolution of publicly available large language models for complex decision-making in breast cancer care

Large language models for precision oncology: Clinical decision support through expert-guided learning.

Large language model (ChatGPT) as a support tool for breast tumor board

Large Language Model Capabilities in Perioperative Risk Prediction and Prognostication

Leveraging Large Language Models for Decision Support in Personalized Oncology

Performance of Retrieval-Augmented Large Language Models to Recommend Head and Neck Cancer Clinical Trials

Large language model use in clinical oncology

Predicting Lung Cancer Patient Prognosis with Large Language Models

Utilizing large language models in breast cancer management: systematic review

Applications of large language models in cancer care: current evidence and future perspectives

Large language models and multimodal foundation models for precision oncology

Exploring Large Language Models for Specialist-level Oncology Care

Large Language Models-Enabled Digital Twins for Precision Medicine in Rare Gynecological Tumors

Performance of a trained large language model to provide clinical trial recommendation in a head and neck cancer population.

Enhancing Biomarker-Based Oncology Trial Matching Using Large Language Models

Demystifying Large Language Models for Medicine: A Primer

Uncertainty Quantification for Clinical Outcome Predictions with (Large) Language Models

Large language models as decision aids in neuro-oncology: a review of shared decision-making applications

Large language models improve clinical decision making of medical students through patient simulation and structured feedback: a randomized controlled trial