Abstract:A significant challenge in analyzing spatial transcriptomics data is the effective and efficient detection of spatially variable genes (SVGs), whose expression exhibits non-random spatial patterns in tissues. Many SVGs show spatial variation in expression that is highly correlated with cell type categories or compositions, leading to the concept of cell type-specific spatially variable genes (ctSVGs). Existing statistical methods for detecting ctSVGs treat cell type-specific spatial effects as fixed effects when modeling, resulting in a critical issue: the testing results are not invariant to the rotation of spatial coordinates. Additionally, an SVG may display random spatial patterns within a cell type, and a ctSVG may exhibit random spatial patterns from a general perspective, indicating that an SVG does not necessarily have to be a ctSVG, and vice versa. This poses challenges in real analysis when detecting SVGs or ctSVGs. To address these problems, we propose STANCE, a unified statistical model developed to detect both SVG and ctSVGs in spatial transcriptomics. By integrating gene expression, spatial location, and cell type composition through a linear mixed-effect model, STANCE enables the identification of both SVGs and ctSVGs in an initial stage, followed by a second stage test dedicated to ctSVG detection. Its design ensures robustness in complex scenarios and the results are spatial rotation invariant. We demonstrated the performance of STANCE through comprehensive simulations and analyses of three public datasets. The downstream analyses based on ctSVGs detected by STANCE suggest promising future applications of the model in spatial transcriptomics and various areas of genome biology. A software implementation of STANCE is available at https://github.com/Cui-STT-Lab/STANCE.

What problem does this paper attempt to address?

The main problems that this paper attempts to solve are two key issues existing in the current methods when detecting cell - type - specific spatial variant genes (ctSVGs) in spatial transcriptomics data: 1. **Rotation invariance problem**: The existing statistical methods treat cell - type - specific spatial effects as fixed effects during the modeling process, resulting in detection results that are not rotation - invariant with respect to spatial coordinates. This means that if the tissue sample is observed at different angles, the detection results may vary greatly, leading to a high false - positive or false - negative rate. This problem is particularly prominent in actual analysis because tissue sections are often randomly positioned during sample preparation, and different directions will give different spatial coordinates. 2. **Complex relationship between SVG and ctSVG**: There is a complex inter - relationship between spatial variant genes (SVGs) and cell - type - specific spatial variant genes (ctSVGs). An SVG may show a random spatial pattern within any cell type, and a ctSVG may also show a random spatial pattern as a whole. This indicates that an SVG is not necessarily a ctSVG, and vice versa. This complex relationship poses challenges for SVG or ctSVG detection in actual analysis. To solve these problems, the paper proposes **STANCE** (Spatial Transcriptomics ANalysis of genes with Cell - type - specific Expression), which is a unified statistical model aimed at simultaneously detecting SVGs and ctSVGs in spatial transcriptomics data. By integrating gene expression, spatial location and cell - type composition information and using a linear mixed - effects model, STANCE can identify SVGs and ctSVGs in the initial stage, and then specifically detect ctSVGs in the second stage. Its design ensures robustness in complex scenarios, and the results are invariant to rotation and transformation of spatial coordinates.

STANCE: a unified statistical model to detect cell-type-specific spatially variable genes in spatial transcriptomics

PROST: quantitative identification of spatially variable genes and domain detection in spatial transcriptomics

Spanve: an Statistical Method to Detect Clustering-friendly Spatially Variable Genes in Large-scale Spatial Transcriptomics Data

SPACE: Spatially variable gene clustering adjusting for cell type effect for improved spatial domain detection

STAIG: Spatial Transcriptomics Analysis via Image-Aided Graph Contrastive Learning for Domain Exploration and Alignment-Free Integration

Uncover spatially informed variations for single-cell spatial transcriptomics with STew

Statistical batch-aware embedded integration, dimension reduction and alignment for spatial transcriptomics

DSTG: Deconvoluting Spatial Transcriptomics Data Through Graph-Based Artificial Intelligence

DESpace : spatially variable gene detection via differential expression testing of spatial clusters

Detecting anomalous anatomic regions in spatial transcriptomics with STANDS

TIST: Transcriptome and Histopathological Image Integrative Analysis for Spatial Transcriptomics

ST-SCSR: identifying spatial domains in spatial transcriptomics data via structure correlation and self-representation

Accurate, Scalable and Cross-platform Cell Identification for High-resolution Spatial Transcriptomics

Categorization of 33 computational methods to detect spatially variable genes from spatially resolved transcriptomics data

Cell-specific priors rescue differential gene expression in spatial spot-based technologies

Probabilistic cell/domain-type assignment of spatial transcriptomics data with SpatialAnno

High-density generation of spatial transcriptomics with STAGE

Benchmarking clustering, alignment, and integration methods for spatial transcriptomics

Alignment and Integration of Spatial Transcriptomics Data

Interpretable spatially aware dimension reduction of spatial transcriptomics with STAMP