Abstract:Neoadjuvant chemotherapy (NAC) is the standard of care for selected patients with high-risk early-stage breast cancer with pathologic complete response (pCR) being the most prominent predictor of favorable outcomes. Here, we sought to study the predictive capacity of integrating orthogonal diagnostic measures on predicting pCR relative to standard clinicopathologic features. We developed a computational model integrating radiology and pathology images, and tumor genomics to automatically predict pCR from multimodal data. We present an interim analysis on a cohort of 957 patients with at least one available pre-NAC data modality. The baseline AUC for pCR prediction by a trained and tested logistic regression model on 857 patients using standard clinicopathologic features including receptor subtype, demographic information, and stage was 0.77. MR images were input into a convolutional neural network (CNN) and a radiomics model. The trained CNN and radiomics models using selected images of 576 patients with pre-NAC MR images achieved AUCs of 0.65 and 0.60 on 164 hold-out test cases, respectively. We trained a multiple instance learning-based weakly supervised learning (MIL-WSL) model using 537,762 extracted tiles from whole slide images (WSI) of digital histopathology scans from 522 patients. The MIL-WSL model achieved AUC of 0.63 for pCR prediction on a hold-out test set of pre-NAC biopsies from 239 patients. A feature based classifier trained on 76 cases using tumor genomic features such as mutational burden, microsatellite instability, fraction genome altered, ploidy, purity, mutation and copy number alterations in selected genes achieved an AUC of 0.72 on 83 hold-out test cases. We then combined unimodal radiology, histopathology, and genomic predictions in a deterministic manner. This multimodal combination on an independent 68-patient test set achieved an AUC of 0.84, indicating increased power to resolve pCR than any modality alone, and over clinicopathologic baseline. Together, we present approaches to train models end-to-end using tensor fusion networks and attention-gating combined with MIL. Automated multimodal methods are here shown to improve prediction over established clinical parameters alone, motivating our ongoing efforts to refine and improve the model so as to achieve higher levels of efficiency. We anticipate these interim results will be further improved through refinement of input features and increasing the number of patients included in the final validation cohort. Citation Format: Pegah Khosravi, Elizabeth J. Sutton, Justin Jee, Timothy Dalfonso, Christopher J. Fong, Doori Rose, Edaise M. Da Silva, Armaan Kohli, David Joon Ho, Mehnaj S. Ahmed, Danny Martinez, Anika Begum, Elizabeth Zakszewski, Andrew Aukerman, Yanis Tazi, Katja Pinker-Domenig, Sarah Eskreis-Winkler, Atif J. Khan, Edi Brogi, Elizabeth Morris, Sarat Chandarlapaty, George Plitas, Simon Powell, Monica Morrow, Larry Norton, Jianjiong Gao, Mark Robson, Hong Zhang, Sohrab Shah, Pedram Razavi, MSK-MIND Consortium. Prediction of neoadjuvant treatment outcomes with multimodal data integration in breast cancer [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl) nr 1928.

Multimodal modeling with low-dose CT and clinical information for diagnostic artificial intelligence on mediastinal tumors: a preliminary study

Multimodal Integration of Longitudinal Noninvasive Diagnostics for Survival Prediction in Immunotherapy Using Deep Learning

Medical transformer for multimodal survival prediction in intensive care: integration of imaging and non-imaging data

Multimodal fully automated predictive model for therapeutic efficacy of first-line cancer immunotherapy based on clinical information and imaging modalities including brain MRI and chest CT images in advanced non-small cell lung cancer.

Multimodal MRI Deep Learning for Predicting Central Lymph Node Metastasis in Papillary Thyroid Cancer

Machine learning models from computed tomography to diagnose thymic epithelial tumors requiring combined resection

Multimodal AI Combining Clinical and Imaging Inputs Improves Prostate Cancer Detection

Abstract 4905: Multimodal transformer model improves survival prediction in lung cancer compared to unimodal approaches

A transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics

A deep learning model to enhance the classification of primary bone tumors based on incomplete multimodal images in X-ray, CT, and MRI

Fusion of CT images and clinical variables based on deep learning for predicting invasiveness risk of stage I lung adenocarcinoma.

Multi-center evaluation of artificial intelligent imaging and clinical models for predicting neoadjuvant chemotherapy response in breast cancer

Therapeutic Decision Making in Prevascular Mediastinal Tumors Using CT Radiomics and Clinical Features: Upfront Surgery or Pretreatment Needle Biopsy?

Vision Transformer-based Multimodal Feature Fusion Network for Lymphoma Segmentation on PET/CT Images

Multimodal Ultrasound Radiomic Technology for Diagnosing Benign and Malignant Thyroid Nodules of Ti-Rads 4-5: A Multicenter Study

Longitudinal Multimodal Transformer Integrating Imaging and Latent Clinical Signatures From Routine EHRs for Pulmonary Nodule Classification

Multimodal Artificial Intelligence in Medicine

Abstract 1928: Prediction of neoadjuvant treatment outcomes with multimodal data integration in breast cancer

Leveraging Transformers to Improve Breast Cancer Classification and Risk Assessment with Multi-modal and Longitudinal Data

Abstract 2313: Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images