Abstract:Abstract Motivation Cell-type annotation of single-cell RNA-sequencing (scRNA-seq) data is a hallmark of biomedical research and clinical application. Current annotation tools usually assume the simultaneous acquisition of well-annotated data, but without the ability to expand knowledge from new data. Yet, such tools are inconsistent with the continuous emergence of scRNA-seq data, calling for a continuous cell-type annotation model. In addition, by their powerful ability of information integration and model interpretability, transformer-based pre-trained language models have led to breakthroughs in single-cell biology research. Therefore, the systematic combining of continual learning and pre-trained language models for cell-type annotation tasks is inevitable. Results We herein propose a universal cell-type annotation tool, called CANAL, that continuously fine-tunes a pre-trained language model trained on a large amount of unlabeled scRNA-seq data, as new well-labeled data emerges. CANAL essentially alleviates the dilemma of catastrophic forgetting, both in terms of model inputs and outputs. For model inputs, we introduce an experience replay schema that repeatedly reviews previous vital examples in current training stages. This is achieved through a dynamic example bank with a fixed buffer size. The example bank is class-balanced and proficient in retaining cell-type-specific information, particularly facilitating the consolidation of patterns associated with rare cell types. For model outputs, we utilize representation knowledge distillation to regularize the divergence between previous and current models, resulting in the preservation of knowledge learned from past training stages. Moreover, our universal annotation framework considers the inclusion of new cell types throughout the fine-tuning and testing stages. We can continuously expand the cell-type annotation library by absorbing new cell types from newly arrived, well-annotated training datasets, as well as automatically identify novel cells in unlabeled datasets. Comprehensive experiments with data streams under various biological scenarios demonstrate the versatility and high model interpretability of CANAL. Availability An implementation of CANAL is available from https://github.com/aster-ww/CANAL-torch. Contact dengmh@pku.edu.cn Supplementary information Supplementary data are available at Journal Name online.

Exploring Parameter-Efficient Fine-Tuning of a Large-Scale Pre-Trained Model for Scrna-Seq Cell Type Annotation

Parameter-Efficient Fine-Tuning Enhances Adaptation of Single Cell Large Language Model for Cell Type Identification

scPretrain: multi-task self-supervised learning for cell-type classification

scBERT as a large-scale pretrained deep language model for cell type annotation of single-cell RNA-seq data

Parameter-efficient fine-tuning of large-scale pre-trained language models

Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

Evaluation of some aspects in supervised cell type identification for single-cell RNA-seq: classifier, feature selection, and reference construction

Scellseg: a Style-Aware Cell Instance Segmentation Tool with Pre-Training and Contrastive Fine-Tuning

Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models

Scplan: a Hierarchical Computational Framework for Single Transcriptomics Data Annotation, Integration and Cell-Type Label Refinement.

scInterpreter: Training Large Language Models to Interpret scRNA-seq Data for Cell Type Annotation

An Empirical Study of Parameter-Efficient Fine-Tuning Methods for Pre-Trained Code Models.

Fine-tuning large neural language models for biomedical natural language processing

Arbitrary Few Parameters Are Good Enough for Adapting Large-scale Pre-trained Language Models

Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

Making Parameter-efficient Tuning More Efficient: A Unified Framework for Classification Tasks.

Large-scale foundation model on single-cell transcriptomics

scRCA: a Siamese network-based pipeline for the annotation of cell types using imperfect single-cell RNA-seq reference data

scSwinTNet: A Cell Type Annotation Method for Large-Scale Single-Cell RNA-Seq Data Based on Shifted Window Attention

A self-training interpretable cell type annotation framework using specific marker gene

Continually adapting pre-trained language model to universal annotation of single-cell RNA-seq data