Abstract:533 Background: Accurate prediction of response to neoadjuvant chemotherapy (NAC) is essential for optimizing treatment outcomes in muscle-invasive bladder cancer (MIBC). We present a cutting-edge multimodal deep learning model aimed at predicting outcomes of NAC from accessible H & E images and molecular data. Methods: We designed a Graph-based Multi-modal Late Fusion (GMLF) deep learning model that integrates three types of data from the S1314-COXEN clinical trial: 1) Neural embeddings from 182 whole slide images (WSIs), generated via the ResNet50 architecture. 2) Cell types based on morphology, including cancer, necrotic, immune, and stromal cells, and their spatial positions within the tumor microenvironment extracted from WSIs using the HoVer-Net framework, a multiple-branch convolutional neural network for predicting cell types. 3) Patient-level RNA expression data derived from 1,071-dimensional gene expression vectors. The dataset was randomly divided into 80% for model training, using 5-fold cross-validation (5-fold CV), and 20% for internal validation. Results: Our dataset included 182 WSIs of 180 patients who were randomized to receive either gemcitabine-cisplatin or dose-dense methotrexate-vinblastine-adriamycin-doxorubicin-cisplatin. Of all patients, 30.8% demonstrated a complete pathological response (pT0 after radical cystectomy). Our GMLF model achieved an AUROC of 0.7417 ± 0.1021 for predicting whether a patient showed a complete pathologic response to NAC in a 5-fold CV. Using an 80/20 training and validation split, the model yielded an AUROC of 0.7236. Interestingly, our model emphasized the contribution of spatial information collected from WSIs. For selecting the best architecture for the histology branch, the Slidegraph+ framework, a graph neural network-based model, achieved an AUROC of 0.6938 ± 0.0565, significantly outperforming patch-level models as the second-best, which achieved AUROC of 0.5572 ± 0.1793. Shapley Additive Explanation (SHAP) analysis revealed RNA expression branch as the most influential, with a mean SHAP magnitude of 0.13, followed by the neural embeddings at 0.12. SHAP-based interpretation identified the most important expressed genes impacting model predictability, including TP63. Our gene set analysis identified the most influential pathways for predicting response, which included basal differentiation and myo-fibroblast enrichment (p-value < 0.05). Conclusions: Our interpretable GMLF model accurately predicts NAC response in patients with MIBC by integrating standard H&E images and RNA expression data. Our model interpretations not only revealed the importance of each modality but also uncovered histopathological, cellular, and molecular underpinnings of response to NAC in MIBC. These findings open the door to AI-guided development of precision therapies for MIBC. Clinical trial information: NCT02177695 .

A transfer learning-based graph neural network model using mIF images to predict neoadjuvant immunochemotherapy response in patients with gastrointestinal cancer.

An Immune-Related Gene Signature for Predicting Neoadjuvant Chemoradiotherapy Efficacy in Rectal Carcinoma

Radiomics Model Based on Intratumoral and Peritumoral Features for Predicting Major Pathological Response in Non-Small Cell Lung Cancer Receiving Neoadjuvant Immunochemotherapy

Enhancing Prediction for Tumor Pathologic Response to Neoadjuvant Immunochemotherapy in Locally Advanced Esophageal Cancer by Dynamic Parameters from Clinical Assessments

Cancer immunotherapy response prediction from multi-modal clinical and image data using semi-supervised deep learning

TG468: A Text Graph Convolutional Network for Predicting Clinical Response to Immune Checkpoint Inhibitor Therapy

Immunotherapy Efficacy Prediction for Non-small Cell Lung Cancer Using Multi-view Adaptive Weighted Graph Convolutional Networks

Computed tomography-based radiomic model for the prediction of neoadjuvant immunochemotherapy response in patients with advanced gastric cancer

Gene expression-based machine learning approach to predict immunotherapy response and survival in patients with advanced cancer.

Predicting gastric cancer response to anti-HER2 therapy or anti-HER2 combined immunotherapy based on multi-modal data

Predicting therapeutic response to neoadjuvant immunotherapy based on an integration model in resectable stage IIIA (N2) non-small cell lung cancer

CT-based radiomics analysis in the prediction of response to neoadjuvant chemotherapy in locally advanced gastric cancer: A dual-center study

Non-invasive CT Imaging Biomarker to Predict Immunotherapy Response in Gastric Cancer: a Multicenter Study

To develop a prognostic model for neoadjuvant immunochemotherapy efficacy in esophageal squamous cell carcinoma by analyzing the immune microenvironment

Predicting response to immunotherapy in gastric cancer via multi-dimensional analyses of the tumour immune microenvironment

Predicting clinical outcomes in the S1314-COXEN trial using a multimodal deep learning model integrating histopathology, cell types, and gene expression.

A high interferon gamma signature of CD8+ T cells predicts response to neoadjuvant immunotherapy plus chemotherapy in gastric cancer

An integrated radiopathomics machine learning model to predict pathological response to preoperative chemotherapy in gastric cancer

Machine learning models based on immunological genes to predict the response to neoadjuvant therapy in breast cancer patients

A Deep Learning Model to Predict the Response to Neoadjuvant Chemoradiotherapy by the Pretreatment Apparent Diffusion Coefficient Images of Locally Advanced Rectal Cancer

Computed Tomography-Based Radiomics Analysis for Prediction of Response to Neoadjuvant Chemotherapy in Breast Cancer Patients