Abstract:Over 150 000 Americans are diagnosed with colorectal cancer (CRC) every year, and annually over 50 000 individuals will die from CRC, necessitating improvements in screening, prognostication, disease management, and therapeutic options. Tumor metastasis is the primary factor related to the risk of recurrence and mortality. Yet, screening for nodal and distant metastasis is costly, and invasive and incomplete resection may hamper adequate assessment. Signatures of the tumor-immune microenvironment (TIME) at the primary site can provide valuable insights into the aggressiveness of the tumor and the effectiveness of various treatment options. Spatially resolved transcriptomics technologies offer an unprecedented characterization of TIME through high multiplexing, yet their scope is constrained by cost. Meanwhile, it has long been suspected that histological, cytological, and macroarchitectural tissue characteristics correlate well with molecular information (e.g., gene expression). Thus, a method for predicting transcriptomics data through inference of RNA patterns from whole slide images (WSI) is a key step in studying metastasis at scale. In this work, we collected tissue from 4 stage-III (pT3) matched colorectal cancer patients for spatial transcriptomics profiling. The Visium spatial transcriptomics (ST) assay was used to measure transcript abundance for 17 943 genes at up to 5000 55-micron (i.e., 1-10 cells) spots per patient sampled in a honeycomb pattern, co-registered with hematoxylin and eosin (H&E) stained WSI. The Visium ST assay can measure expression at these spots through tissue permeabilization of mRNAs, which are captured through spatially (i.e., x-y positional coordinates) barcoded, gene specific oligo probes. WSI subimages were extracted around each co-registered Visium spot and were used to predict the expression at these spots using machine learning models. We prototyped and compared several convolutional, transformer, and graph convolutional neural networks to predict spatial RNA patterns at the Visium spots under the hypothesis that the transformer- and graph-based approaches better capture relevant spatial tissue architecture. We further analyzed the model's ability to recapitulate spatial autocorrelation statistics using SPARK and SpatialDE. Overall, the results indicate that the transformer- and graph-based approaches were unable to outperform the convolutional neural network architecture, though they exhibited optimal performance for relevant disease-associated genes. Initial findings suggest that different neural networks that operate on different scales are relevant for capturing distinct disease pathways (e.g., epithelial to mesenchymal transition). We add further evidence that deep learning models can accurately predict gene expression in whole slide images and comment on understudied factors which may increase its external applicability (e.g., tissue context). Our preliminary work will motivate further investigation of inference for molecular patterns from whole slide images as metastasis predictors and in other applications.

MOSBY enables multi-omic inference and spatial biomarker discovery from whole slide images

Spatial transcriptomics inferred from pathology whole-slide images links tumor heterogeneity to survival in breast and lung cancer

mosna reveals different types of cellular interactions predictive of response to immunotherapies and survival in cancer

Spatial transcriptomics using combinatorial fluorescence spectral and lifetime encoding, imaging and analysis

Mapping Cells Through Time and Space with Moscot

Spatial Omics Driven Crossmodal Pretraining Applied to Graph-based Deep Learning for Cancer Pathology Analysis

MOSAIC: Multi-Omic Spatial Atlas in Cancer, effect on precision oncology.

Enhancing Spatial Transcriptomics Analysis by Integrating Image-Aware Deep Learning Methods

A deep learning-based multiscale integration of spatial omics with tumor morphology.

Development of Multiomics in Situ Pairwise Sequencing (Mip-Seq) for Single-cell Resolution Multidimensional Spatial Omics

Multimodal Outer Arithmetic Block Dual Fusion of Whole Slide Images and Omics Data for Precision Oncology

Spatial Immunophenotyping from Whole-Slide Multiplexed Tissue Imaging Using Convolutional Neural Networks

Inferring single-cell spatial gene expression with tissue morphology via explainable deep learning

SpatialCells: automated profiling of tumor microenvironments with spatially resolved multiplexed single-cell data

Inferring spatial transcriptomics markers from whole slide images to characterize metastasis-related spatial heterogeneity of colorectal tumors: A pilot study

Spatially Aware Graph Neural Networks and Cross-Level Molecular Profile Prediction in Colon Cancer Histopathology: a Retrospective Multi-Cohort Study.

SM-Omics is an automated platform for high-throughput spatial multi-omics

Machine learning enabled prediction of digital biomarkers from whole slide histopathology images

Path2Space: An AI Approach for Cancer Biomarker Discovery Via Histopathology Inferred Spatial Transcriptomics

Mosaic integration of spatial multi-omics with SpaMosaic