Abstract:Recent advances in measuring high-dimensional modalities, including protein levels and DNA accessibility, at the single-cell level have prompted the need for frameworks capable of handling multi-modal data while simultaneously addressing multiple tasks. Despite these ad- vancements, much of the work in the single-cell domain remains limited, often focusing on either a single-modal or single-task perspective. A few recent studies have ventured into multi- modal, multi-task learning, but we identified a (1) OptimizationConflict issue, leading to suboptimal results when integrating additional modalities, which is undesirable. Furthermore, there is a (2) Costly Interpretability challenge, as current approaches predominantly rely on costly post-hoc methods like SHAP. Motivated by these challenges, we introduce scMoE1, a novel framework that, for the first time, applies Sparse Mixture-of-Experts (SMoE) within the single-cell domain. This is achieved by incorporating an SMoE layer into a transformer block with a cross-attention module. Thanks to its design, scMoE inherently possesses mechanistic interpretability, a critical aspect for understanding underlying mechanisms when handling biological data. Furthermore, from a post-hoc perspective, we enhance interpretability by extending the concept of activation vectors (CAVs). Extensive experiments on simulated datasets, such as Dyngen, and real-world multi-modal single-cell datasets, including {DBiT-seq, Patch-seq, ATAC-seq}, demonstrate the effectiveness of scMoE. Source code of scMoE is available at: https: //github.com/UNITES-Lab/scMoE. ### Competing Interest Statement The authors have declared no competing interest.

ACE: a versatile contrastive learning framework for single-cell mosaic integration

Con-AAE: contrastive cycle adversarial autoencoders for single-cell multi-omics alignment and integration

MIDAS: a deep generative model for mosaic integration and knowledge transfer of single-cell multimodal data

Single-cell mosaic integration and cell state transfer with auto-scaling self-attention mechanism

Scmoe: Single-Cell Multi-Modal Multi-Task Learning Via Sparse Mixture-of-Experts

Contrastively generative self-expression model for single-cell and spatial multimodal data

Multi-modal contrastive learning of subcellular organization using DICE

Integrating T-cell Receptor and Transcriptome for Large-Scale Single-Cell Immune Profiling Analysis

Neurologic outcome in patients with MRI pattern of damage typical for neonatal hypoglycemia

Comprehensive View Embedding Learning for Single-Cell Multimodal Integration

scAMACE: Model-based approach to the joint analysis of single-cell data on chromatin accessibility, gene expression and methylation

Erbium-doped photonic crystal fiber: Fabrication and its gaining characteristics

scMODAL: A general deep learning framework for comprehensive single-cell multi-omics data alignment with feature links

Unlocking cross-modal interplay of single-cell joint profiling with CellMATE

Unlocking cross-modal interplay of single-cell and spatial joint profiling with CellMATE

Integrated analysis of multimodal single-cell data with structural similarity

scCobra: Contrastive cell embedding learning with domain-adaptation for single-cell data integration and harmonization

Mosaic integration of spatial multi-omics with SpaMosaic

MXM-CLR: A Unified Framework for Contrastive Learning of Multifold Cross-Modal Representations

Deep cross-omics cycle attention model for joint analysis of single-cell multi-omics data

scButterfly: a versatile single-cell cross-modality translation method via dual-aligned variational autoencoders