Abstract:The rapid advances in deep learning-based computational pathology and radiology have demonstrated the promise of using whole slide images (WSIs) and radiology images for survival prediction in cancer patients. However, most image-based survival prediction methods are limited to using either histology or radiology alone, leaving integrated approaches across histology and radiology relatively underdeveloped. There are two main challenges in integrating WSIs and radiology images: (1) the gigapixel nature of WSIs and (2) the vast difference in spatial scales between WSIs and radiology images. To address these challenges, in this work, we propose an interpretable, weakly-supervised, multimodal learning framework, called Hierarchical Multimodal Co-Attention Transformer (HMCAT), to integrate WSIs and radiology images for survival prediction. Our approach first uses hierarchical feature extractors to capture various information including cellular features, cellular organization, and tissue phenotypes in WSIs. Then the hierarchical radiology-guided co- attention (HRCA) in HMCAT characterizes the multimodal interactions between hierarchical histology-based visual concepts and radiology features and learns hierarchical co- attention mappings for two modalities. Finally, HMCAT combines their complementary information into a multimodal risk score and discovers prognostic features from two modalities by multimodal interpretability. We apply our approach to two cancer datasets (365 WSIs with matched magnetic resonance [MR] images and 213 WSIs with matched computed tomography [CT] images). Our results demonstrate that the proposed HMCAT consistently achieves superior performance over the unimodal approaches trained on either histology or radiology data alone, as well as other state-of-the-art methods.

Multimodal Cross-Task Interaction for Survival Analysis in Whole Slide Pathological Images

MGCT: Mutual-Guided Cross-Modality Transformer for Survival Outcome Prediction using Integrative Histopathology-Genomic Features

Generating Hypergraph-Based High-Order Representations of Whole-Slide Histopathological Images for Survival Prediction

Multimodal Survival Ensemble Network: Integrating Genomic and Histopathological Insights for Enhanced Cancer Prognosis.

Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images

Multimodal Optimal Transport-based Co-Attention Transformer with Global Structure Consistency for Survival Prediction

Survival Prediction Via Hierarchical Multimodal Co-Attention Transformer: A Computational Histology-Radiology Solution.

Cross-Modal Translation and Alignment for Survival Analysis

Multi-instance Multi-task Learning for Joint Clinical Outcome and Genomic Profile Predictions from the Histopathological Images

Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction

Pathology-and-genomics Multimodal Transformer for Survival Outcome Prediction

Characterizing the Survival-Associated Interactions between Tumor-infiltrating Lymphocytes and Tumors from Pathological Images and Multi-omics Data

Hierarchical Transformer for Survival Prediction Using Multimodality Whole Slide Images and Genomics

Multimodal Prototyping for cancer survival prediction

Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networks

SurvMamba: State Space Model with Multi-grained Multi-modal Interaction for Survival Prediction

TransSurv: Transformer-Based Survival Analysis Model Integrating Histopathological Images and Genomic Data for Colorectal Cancer.

Pathology-genomic fusion via biologically informed cross-modality graph learning for survival analysis

Look a Group at Once: Multi-Slide Modeling for Survival Prediction

Orchestrating Information Across Tissues Via a Novel Multitask GAT Framework to Improve Quantitative Gene Regulation Relation Modeling for Survival Analysis.

MFCSA-CAT: a Multimodal Fusion Method for Cancer Survival Analysis Based on Cross-Attention Transformer