Abstract:Abstract Digital analysis of pathology whole-slide images is fast becoming a game changer in cancer diagnosis and treatment. Specifically, deep learning methods have shown great potential to support pathology analysis, with recent studies identifying molecular traits that were not previously recognized in pathology H&E whole-slide images. Simultaneous to these developments, it is becoming increasingly evident that tumor heterogeneity is an important determinant of cancer prognosis and susceptibility to treatment, and should therefore play a role in the evolving practices of matching treatment protocols to patients. State of the art diagnostic procedures, however, do not provide automated methods for characterizing and/or quantifying tumor heterogeneity, certainly not in a spatial context. Further, existing methods for analyzing pathology whole-slide images from bulk measurements require many training samples and complex pipelines. Our work addresses these two challenges. First, we train deep learning models to spatially resolve bulk mRNA and miRNA expression levels on pathology whole-slide images (WSIs). Our models reach up to 0.95 AUC on held-out test sets from two cancer cohorts using a simple training pipeline and a small number of training samples. Using the inferred gene expression levels, we further develop a method to spatially characterize tumor heterogeneity. Specifically, we produce tumor molecular cartographies and heterogeneity maps of WSIs and formulate a heterogeneity index (HTI) that quantifies the level of heterogeneity within these maps. Applying our methods to breast and lung cancer slides, we show a significant statistical link between heterogeneity and survival. Our methods potentially open a new and accessible approach to investigating tumor heterogeneity and other spatial molecular properties and their link to clinical characteristics, including treatment susceptibility and survival.

Towards Unified Molecule-Enhanced Pathology Image Representation Learning via Integrating Spatial Transcriptomics

An Image-enhanced Molecular Graph Representation Learning Framework

Unified Modeling Enhanced Multimodal Learning for Precision Neuro-Oncology

Towards a general-purpose foundation model for computational pathology

Multimodal Survival Ensemble Network: Integrating Genomic and Histopathological Insights for Enhanced Cancer Prognosis.

Spatial Omics Driven Crossmodal Pretraining Applied to Graph-based Deep Learning for Cancer Pathology Analysis

Gene-Induced Multimodal Pre-training for Image-Omic Classification

PathOmCLIP: Connecting tumor histology with spatial gene expression via locally enhanced contrastive learning of Pathology and Single-cell foundation model

Spatial transcriptomics inferred from pathology whole-slide images links tumor heterogeneity to survival in breast and lung cancer

Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis

CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology

A General-Purpose Self-Supervised Model for Computational Pathology

StereoMM: A Graph Fusion Model for Integrating Spatial Transcriptomic Data and Pathological Images

MGCT: Mutual-Guided Cross-Modality Transformer for Survival Outcome Prediction using Integrative Histopathology-Genomic Features

Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction

HE2Gene: Image-to-RNA Translation Via Multi-Task Learning for Spatial Transcriptomics Data

Deep Biological Pathway Informed Pathology-Genomic Multimodal Survival Prediction

Exploit Spatially Resolved Transcriptomic Data to Infer Cellular Features from Pathology Imaging Data}

PathoDuet: Foundation Models for Pathological Slide Analysis of H&E and IHC Stains

Transcriptomics-guided Slide Representation Learning in Computational Pathology

Pathology-genomic fusion via biologically informed cross-modality graph learning for survival analysis