Abstract:Abstract Background Single-cell transcriptomics is rapidly advancing our understanding of the cellular composition of complex tissues and organisms. A major limitation in most analysis pipelines is the reliance on manual annotations to determine cell identities, which are time-consuming and irreproducible. The exponential growth in the number of cells and samples has prompted the adaptation and development of supervised classification methods for automatic cell identification. Results Here, we benchmarked 22 classification methods that automatically assign cell identities including single-cell-specific and general-purpose classifiers. The performance of the methods is evaluated using 27 publicly available single-cell RNA sequencing datasets of different sizes, technologies, species, and levels of complexity. We use 2 experimental setups to evaluate the performance of each method for within dataset predictions (intra-dataset) and across datasets (inter-dataset) based on accuracy, percentage of unclassified cells, and computation time. We further evaluate the methods’ sensitivity to the input features, number of cells per population, and their performance across different annotation levels and datasets. We find that most classifiers perform well on a variety of datasets with decreased accuracy for complex datasets with overlapping classes or deep annotations. The general-purpose support vector machine classifier has overall the best performance across the different experiments. Conclusions We present a comprehensive evaluation of automatic cell identification methods for single-cell RNA sequencing data. All the code used for the evaluation is available on GitHub ( https://github.com/tabdelaal/scRNAseq_Benchmark ). Additionally, we provide a Snakemake workflow to facilitate the benchmarking and to support the extension of new methods and new datasets.

The shaky foundations of simulating single-cell RNA sequencing data

scRNA-seq mixology: towards better benchmarking of single cell RNA-seq analysis methods

Benchmarking single cell RNA-sequencing analysis pipelines using mixture control experiments

Systematic evaluation with practical guidelines for single-cell and spatially resolved transcriptomics data simulation under multiple scenarios

Systematic comparative analysis of single cell RNA-sequencing methods

A multicenter study benchmarking single-cell RNA sequencing technologies using reference samples

Beyond benchmarking and towards predictive models of dataset-specific single-cell RNA-seq pipeline performance

Benchmarking UMI-based single-cell RNA-seq preprocessing workflows

SimCH: simulation of single-cell RNA sequencing data by modeling cellular heterogeneity at gene expression level

Bias, robustness and scalability in differential expression analysis of single-cell RNA-seq data

Comprehensive Evaluation of Noise Reduction Methods for Single-Cell RNA Sequencing Data

A benchmark of batch-effect correction methods for single-cell RNA sequencing data

A systematic evaluation of single-cell RNA-sequencing imputation methods

scRNABatchQC: multi-samples quality control for single cell RNA-seq data

Multi-task benchmarking of spatially resolved gene expression simulation models

The impact of package selection and versioning on single-cell RNA-seq analysis

IBRAP: Integrated Benchmarking Single-cell RNA-sequencing Analytical Pipeline

Tools for the Analysis of High-Dimensional Single-Cell RNA Sequencing Data

A comparison of automatic cell identification methods for single-cell RNA sequencing data

Evaluating Imputation Methods for Single-Cell RNA-seq Data

scReadSim: a single-cell RNA-seq and ATAC-seq read simulator