Abstract:Spatial omics and multiplexed imaging technologies provide unprecedented spatial context to characterize molecular variation in complex tissues. Developing unsupervised computational pipeline in a discovery mode is of vital importance to interpret spatially resolved data and derive novel biological insights. We develop Stereo, a unified framework leveraging contrastive learning to jointly model molecular and spatial information in a self-supervised manner. Stereo consists of two modules, StereoCell for neighbor-aware cell identity inference and clustering on spatial transcriptomics data, and StereoNiche for biologically-relevant niche identification on spatial proteomics data. StereoCell uses graph attention to aggregate immediate neighbors’ content for each cell while StereoNiche exploits vision Transformer to explicitly relate position coordinates with marker readouts within certain receptive field. We validate StereoCell’s superior utility on both simulated and real transcriptomics datasets of brain tissues acquired by various technologies with different resolutions. StereoCell can operate on all genes, extract layer-specific signature genes at single-cell resolution and identify clear layer-wise structure with state-of-the-art performance. Applied on tumor tissue sections, StereoCell enables detailed characterization of intratumoral heterogeneity and reveals invasive or metastasis behaviors concordant with expert annotations. Evaluated on a multiplexed fluorescence imaging dataset for colorectal cancer (CRC) patients, StereoNiche refines niche assignments, recapitulates classical follicle structure, and validates enriched T cells-macrophages interdigitation. Via self-distillation operation, StereoNiche also enables extracting bounded layout of tertiary lymphoid structure with no supervision, representing classical Crohn’s-like reaction behavior. Local explanations are derived for a tree-based survival model to validate the clinical relevance of identified niches. For mass spectrometry profiled triple-negative breast cancer (TNBC) samples, StereoNiche recognizes differential tumor-immune interactions to separate compartmentalized from mixed pattern. In brief, Stereo is applicable to a variety of spatial technologies, serving as a powerful method to interrogate context-aware biological process and facilitate hierarchical signature discovery in neuroscience and computational pathology.

StereoMM: A Graph Fusion Model for Integrating Spatial Transcriptomic Data and Pathological Images

Revealing Spatial Multimodal Heterogeneity in Tissues with SpaTrio

Position-informed contrastive learning for spatially resolved omics deciphers hierarchical tissue structure at both cellular and niche levels

Multimodal Survival Ensemble Network: Integrating Genomic and Histopathological Insights for Enhanced Cancer Prognosis.

Spatial transcriptomics inferred from pathology whole-slide images links tumor heterogeneity to survival in breast and lung cancer

StereoSiTE: a framework to spatially and quantitatively profile the cellular neighborhood organized iTME

StereoCell enables highly accurate single-cell segmentation for spatial transcriptomics

stMMR: accurate and robust spatial domain identification from spatially resolved transcriptomics with multimodal feature representation

Stmmr: Accurate and Robust Spatial Domain Identification from Spatially Resolved Transcriptomics with Multi-Modal Feature Representation

A deep learning-based multiscale integration of spatial omics with tumor morphology.

A progressive phased attention model fusing histopathology and genomic features for lung cancer staging

Graph attention-based fusion of pathology images and gene expression for prediction of cancer survival

Elucidating tumor heterogeneity from spatially resolved transcriptomics data by multi-view graph collaborative learning

Multi-modal Spatial Clustering for Spatial Transcriptomics Utilizing High-resolution Histology Images

Enhancing Spatial Transcriptomics Analysis by Integrating Image-Aware Deep Learning Methods

Unveiling Tissue Structure and Tumor Microenvironment from Spatially Resolved Transcriptomics by Hypergraph Learning

A Deep Learning Approach for Tissue Spatial Quantification and Genomic Correlations of Histopathological Images

Characterizing the Survival-Associated Interactions between Tumor-infiltrating Lymphocytes and Tumors from Pathological Images and Multi-omics Data

Integrative analysis of pathological images and multi-dimensional genomic data for early-stage cancer prognosis.

STAIG: Spatial Transcriptomics Analysis via Image-Aided Graph Contrastive Learning for Domain Exploration and Alignment-Free Integration

Histologically Resolved Multiomics Enables Precise Molecular Profiling of Human Intratumor Heterogeneity