A Large Language Model Pipeline for Breast Cancer Oncology

Tristen Pool,Dennis Trujillo
2024-06-14
Abstract:Large language models (LLMs) have demonstrated potential in the innovation of many disciplines. However, how they can best be developed for oncology remains underdeveloped. State-of-the-art OpenAI models were fine-tuned on a clinical dataset and clinical guidelines text corpus for two important cancer treatment factors, adjuvant radiation therapy and chemotherapy, using a novel Langchain prompt engineering pipeline. A high accuracy (0.85+) was achieved in the classification of adjuvant radiation therapy and chemotherapy for breast cancer patients. Furthermore, a confidence interval was formed from observational data on the quality of treatment from human oncologists to estimate the proportion of scenarios in which the model must outperform the original oncologist in its treatment prediction to be a better solution overall as 8.2% to 13.3%. Due to indeterminacy in the outcomes of cancer treatment decisions, future investigation, potentially a clinical trial, would be required to determine if this threshold was met by the models. Nevertheless, with 85% of U.S. cancer patients receiving treatment at local community facilities, these kinds of models could play an important part in expanding access to quality care with outcomes that lie, at minimum, close to a human oncologist.
Artificial Intelligence,Computation and Language
What problem does this paper attempt to address?
The main problem this paper attempts to address is enhancing breast cancer treatment planning by using large language models (LLMs) fine-tuned with medical guidelines and historical patient datasets. Specifically, the study aims to improve the accuracy and consistency of adjuvant radiotherapy and chemotherapy treatment recommendations to enhance patient outcomes. ### Background and Problem - **Prevalence of Breast Cancer**: Breast cancer is one of the most common cancers among women worldwide, with approximately 1.5 million new cases each year, accounting for 25% of all female cancer patients. - **Limited Medical Resources**: Many patients cannot access optimal treatment plans due to the scarcity of medical resources, especially in community healthcare settings where 85% of U.S. cancer patients receive treatment. - **Complexity of Treatment Decisions**: Breast cancer treatment decisions are highly complex and require a high level of expertise, which is not always available in community healthcare settings. ### Research Objectives - **Improve Treatment Recommendation Accuracy**: By fine-tuning large language models (such as OpenAI's GPT-3.5 Turbo, Babbage, and DaVinci) using clinical datasets and clinical guidelines, the study aims to improve the accuracy of adjuvant radiotherapy and chemotherapy treatment recommendations. - **Consistency**: Ensure consistency in treatment recommendations, reducing variability due to differences in physician experience. - **Expand Access to High-Quality Care**: By automating parts of the cancer care process, reduce treatment costs, and broaden the scope of treatment considerations, enabling more patients to access high-quality care. ### Methods - **Dataset**: Use the Duke MRI dataset, which includes demographic, genomic, tumor characteristics, treatment, and prognosis information for 922 breast cancer patients. - **Clinical Guidelines Corpus**: Collect clinical guideline texts from organizations such as ASCO and NCCN. - **Fine-Tuning Process**: Use the Langchain prompt engineering pipeline to convert clinical data and guideline texts into question-answer pairs for fine-tuning the LLMs. - **Model Evaluation**: Evaluate the model's classification accuracy using a validation set and analyze the model's performance under different temperature settings. ### Results - **High Classification Accuracy**: The model achieved a classification accuracy of over 0.85 in predicting adjuvant radiotherapy and chemotherapy. - **Comparison with Human Doctors**: By analyzing confidence intervals, it is estimated that the model may outperform human doctors' treatment recommendations in 8.2% to 13.3% of scenarios. ### Significance - **Decision Support**: These models can assist doctors in making more informed and consistent treatment decisions, thereby improving overall patient outcomes. - **Future Research Directions**: To further validate the model's effectiveness, future clinical trials may be necessary. Through these methods, this study provides a potential solution for breast cancer treatment planning, helping to improve the accuracy and consistency of treatment and expand access to high-quality care.