Abstract:The success of large-scale pre-trained language models in the Natural Language Processing (NLP) domain has encouraged their adoption in genomics and single-cell biology. Developing pre-trained models using the rapidly growing single-cell transcriptomic data helps to unravel the intricate language of cells. However, current single-cell pre-trained models primarily focus on learning gene and cell representations from extensive gene expression data, failing to fully comprehend the biological significance of the gene expression patterns and cell types they identify, which leads to limited interpretability and transferability. We propose scKEPLM, a knowledge-enhanced single-cell pre-training language model integrates a biology knowledge graph into the single-cell transcriptome pre-training process. scKEPLM covers over 41 million single-cell RNA sequences and 8.9 million gene relations. Through parallel pre-training of single-cell transcriptome sequences and genetic knowledge, combined with a Gaussian cross-attention mechanism, scKEPLM precisely aligns cell semantics with genetic information, to learn more accurate and comprehensive representations of single-cell transcriptomes. The introduction of knowledge enhancement has improved the identification of important genes in cells by scKEPLM, and greatly enriched the understanding of cell function and disease mechanism. The scKEPLM model has achieved state-of-the-art performance in more than 12 downstream tasks, including gene annotation, cell annotation, and drug response prediction, demonstrating strong generalization and transferability. Further exploration of the model's interpretability demonstrates its adaptability to variations in gene expression patterns within cells under various physiological or pathological conditions.

Scmulan: a Multitask Generative Pre-Trained Language Model for Single-Cell Analysis

Toward mastering the cell language by learning to generate

scELMo: Embeddings from Language Models are Good Learners for Single-cell Data Analysis

scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI

scGPT: toward building a foundation model for single-cell multi-omics using generative AI

scKEPLM: Knowledge enhanced large-scale pre-trained language model for single-cell transcriptomics

scInterpreter: Training Large Language Models to Interpret scRNA-seq Data for Cell Type Annotation

Large-scale foundation model on single-cell transcriptomics

LangCell: Language-Cell Pre-training for Cell Identity Understanding

scMMT: a multi-use deep learning approach for cell annotation, protein prediction and embedding in single-cell RNA-seq data

ChatCell: Facilitating Single-Cell Analysis with Natural Language

CELLama: Foundation Model for Single Cell and Spatial Transcriptomics by Cell Embedding Leveraging Language Model Abilities

Unified Language Model Pre-training for Natural Language Understanding and Generation

Transformer-based Single-Cell Language Model: A Survey

MulCNN: An efficient and accurate deep learning method based on gene embedding for cell type identification in single-cell RNA-seq data

Machine-Created Universal Language for Cross-lingual Transfer

Scact: Accurate Cross-modality Translation Via Cycle-consistent Training from Unpaired Single-cell Data

scMultiGAN: cell-specific imputation for single-cell transcriptomes with multiple deep generative adversarial networks

Comprehensive single-cell RNA-seq analysis using deep interpretable generative modeling guided by biological hierarchy knowledge

MULAN: Multimodal Protein Language Model for Sequence and Structure Encoding

MuLan-Methyl-multiple transformer-based language models for accurate DNA methylation prediction