Abstract:Abstract Background Single-cell transcriptomics is rapidly advancing our understanding of the cellular composition of complex tissues and organisms. A major limitation in most analysis pipelines is the reliance on manual annotations to determine cell identities, which are time-consuming and irreproducible. The exponential growth in the number of cells and samples has prompted the adaptation and development of supervised classification methods for automatic cell identification. Results Here, we benchmarked 22 classification methods that automatically assign cell identities including single-cell-specific and general-purpose classifiers. The performance of the methods is evaluated using 27 publicly available single-cell RNA sequencing datasets of different sizes, technologies, species, and levels of complexity. We use 2 experimental setups to evaluate the performance of each method for within dataset predictions (intra-dataset) and across datasets (inter-dataset) based on accuracy, percentage of unclassified cells, and computation time. We further evaluate the methods’ sensitivity to the input features, number of cells per population, and their performance across different annotation levels and datasets. We find that most classifiers perform well on a variety of datasets with decreased accuracy for complex datasets with overlapping classes or deep annotations. The general-purpose support vector machine classifier has overall the best performance across the different experiments. Conclusions We present a comprehensive evaluation of automatic cell identification methods for single-cell RNA sequencing data. All the code used for the evaluation is available on GitHub ( https://github.com/tabdelaal/scRNAseq_Benchmark ). Additionally, we provide a Snakemake workflow to facilitate the benchmarking and to support the extension of new methods and new datasets.

From G1 to M: a comparative study of methods for identifying cell cycle phases

Identification of Human Cell Cycle Phase Markers Based on Single-Cell RNA-Seq Data by Using Machine Learning Methods

Computational challenges of cell cycle analysis using single cell transcriptomics

Modification of Seurat v4 for the Development of a Phase Assignment Tool Able to Distinguish between G2 and Mitotic Cells

scRNA-seq mixology: towards better benchmarking of single cell RNA-seq analysis methods

Categorization and analysis of 14 computational methods for estimating cell potency from single-cell RNA-seq data

Benchmarking single cell RNA-sequencing analysis pipelines using mixture control experiments

A comprehensive comparison of supervised and unsupervised methods for cell type identification in single-cell RNA-seq

A multicenter study benchmarking single-cell RNA sequencing technologies using reference samples

Systematic comparative analysis of single cell RNA-sequencing methods

Evaluating Imputation Methods for Single-Cell RNA-seq Data

CCPE: cell cycle pseudotime estimation for single cell RNA-seq data

Assessment of batch-correction methods for scRNA-seq data with a new test metric

Beyond benchmarking: towards predictive models of dataset-specific single-cell RNA-seq pipeline performance

The shaky foundations of simulating single-cell RNA sequencing data

Precision and Accuracy of Single-Cell/Nuclei RNA Sequencing Data

A comparison of automatic cell identification methods for single-cell RNA sequencing data

Evaluation of single-cell classifiers for single-cell RNA sequencing data sets

ImmunoCellCycle-ID: A high-precision immunofluorescence-based method for cell cycle identification

A Strategy to Compare Single-Cell RNA Sequencing Data Sets Provides Phenotypic Insight into Cellular Heterogeneity Underlying Biological Similarities and Differences Between Samples

Comparison of Scanpy-based Algorithms to Remove the Batch Effect from Single-Cell RNA-seq Data