Abstract:e15161 Background: Analyzing genetic mutations is essential for personalized therapy decisions in oncology and beyond. In colorectal cancer (CRC), mutations in BRAF ( BRAF mut ) render patients eligible for targeted treatment with BRAF inhibitors and immunotherapy. BRAF mut is routinely analyzed via genomic sequencing, which, however, is costly and does not provide diagnostic results immediately. Reliable detection of BRAF mut status from histological images using artificial intelligence (AI) could provide a more accessible and faster diagnostic alternative. Previous approaches to image-based BRAF mut analysis, however, failed to reach a critical level of accuracy on independent test cohorts, with reported area under the receiver operating characteristic curve (AUROC) values typically below 0.8. Methods: We hypothesized that combining tissue image analysis with clinical patient data could improve diagnostic accuracy for BRAF mut . Following our hypothesis, we developed a multi-modal AI model on n = 426 CRC cases from the TCGA database using BRAF mut data, H&E-stained tissue images, and information on patient parameters that are readily available in clinical practice and that are known to be associated with BRAF mut frequency. The model was based on a vision transformer architecture incorporating pathology-specific information using a pretrained embedding model for encoding tissue image features. Model performance was validated on an independent hold-out TCGA cohort of n = 107 samples and an additional external cohort of n = 104 CRC samples from the CPTAC database. Results: Our multi-modal AI model reached an AUROC of 0.85 ± 0.02 on the TCGA hold-out test set. Notably, on the second, external CPTAC dataset the predictive accuracy level was similar (AUROC of 0.83 ± 0.02) indicating the robustness of our approach to generalize to different datasets. Additional analyses showed that the significant performance boost over previous approaches was gained by the multi-modal analysis setup, as predicting BRAF mut based on either tissue, or clinical data alone led to lower accuracy values. Explainability analysis of AI results confirmed that relevant biological features on the histological slides, mainly tumor areas, were considered by the model for decision making. Conclusions: The combination of H&E images with clinically readily available patient parameters enabled us to develop a model predicting BRAF mut in CRC that surpasses previous approaches in diagnostic accuracy on independent evaluation cohorts. This indicates that combining histological tissue information with clinical parameters can increase the ability of AI models to predict genetic variants from tissue and more generally highlights the potential of multi-modal deep learning approaches to enhance predictive AI models in digital pathology.

Apple Machine Learning Algorithms Successfully Detect Colon Cancer but Fail to Predict KRAS Mutation Status

Using Apple Machine Learning Algorithms to Detect and Subclassify Non-Small Cell Lung Cancer

Google Auto ML versus Apple Create ML for Histopathologic Cancer Diagnosis; Which Algorithms Are Better?

Accurate prediction of colorectal cancer diagnosis using machine learning based on immunohistochemistry pathological images

Noninvasive KRAS mutation estimation in colorectal cancer using a deep learning method based on routine CT imaging

A multitask dual‐stream attention network for the identification of KRAS mutation in colorectal cancer

Noninvasive KRAS mutation estimation in colorectal cancer using a deep learning method based on CT imaging

A multi-modal artificial intelligence model trained on H&E tissue images and clinical patient data for prediction of BRAF mutations in colorectal cancer.

Predictive radiogenomics modeling of EGFR mutation status in lung cancer

KRASFormer: a fully vision transformer-based framework for predicting KRAS gene mutations in histopathological images of colorectal cancer

Identifying Genetic Mutation Status in Patients with Colorectal Cancer Liver Metastases Using Radiomics-Based Machine-Learning Models

Artificial Intelligence in Predicting Microsatellite Instability and KRAS, BRAF Mutations from Whole-Slide Images in Colorectal Cancer: A Systematic Review

A Deep Learning Model Based on MRI and Clinical Factors Facilitates Noninvasive Evaluation of KRAS Mutation in Rectal Cancer

Machine learning for predicting colon cancer recurrence

Deep neural network for the prediction of KRAS, NRAS, and BRAF genotypes in left-sided colorectal cancer based on histopathologic images

Machine Learning-Based Radiomics Signatures for EGFR and KRAS Mutations Prediction in Non-Small-Cell Lung Cancer

Next Generation Radiogenomics Sequencing for Prediction of EGFR and KRAS Mutation Status in NSCLC Patients Using Multimodal Imaging and Machine Learning Approaches

Predicting Kirsten Rat Sarcoma Virus Gene Mutation Status in Patients With Colorectal Cancer by Radiomics Models Based on Multiphasic CT

Machine Learning Model for Predicting Postoperative Survival of Patients with Colorectal Cancer

Colon Cancer Diagnosis Based on Machine Learning and Deep Learning: Modalities and Analysis Techniques

CT-based non-invasive identification of the most common gene mutation status in patients with non-small cell lung cancer