Abstract:Abstract Background Single-cell transcriptomics is rapidly advancing our understanding of the cellular composition of complex tissues and organisms. A major limitation in most analysis pipelines is the reliance on manual annotations to determine cell identities, which are time-consuming and irreproducible. The exponential growth in the number of cells and samples has prompted the adaptation and development of supervised classification methods for automatic cell identification. Results Here, we benchmarked 22 classification methods that automatically assign cell identities including single-cell-specific and general-purpose classifiers. The performance of the methods is evaluated using 27 publicly available single-cell RNA sequencing datasets of different sizes, technologies, species, and levels of complexity. We use 2 experimental setups to evaluate the performance of each method for within dataset predictions (intra-dataset) and across datasets (inter-dataset) based on accuracy, percentage of unclassified cells, and computation time. We further evaluate the methods’ sensitivity to the input features, number of cells per population, and their performance across different annotation levels and datasets. We find that most classifiers perform well on a variety of datasets with decreased accuracy for complex datasets with overlapping classes or deep annotations. The general-purpose support vector machine classifier has overall the best performance across the different experiments. Conclusions We present a comprehensive evaluation of automatic cell identification methods for single-cell RNA sequencing data. All the code used for the evaluation is available on GitHub ( https://github.com/tabdelaal/scRNAseq_Benchmark ). Additionally, we provide a Snakemake workflow to facilitate the benchmarking and to support the extension of new methods and new datasets.

A comparison of automatic cell identification methods for single-cell RNA sequencing data

A comprehensive comparison of supervised and unsupervised methods for cell type identification in single-cell RNA-seq

Systematic comparative analysis of single cell RNA-sequencing methods

Automatic Cell Type Identification Methods for Single-Cell RNA Sequencing.

Evaluation of single-cell classifiers for single-cell RNA sequencing data sets

Evaluation of Classification in Single Cell Atac-Seq Data with Machine Learning Methods

Evaluation of some aspects in supervised cell type identification for single-cell RNA-seq: classifier, feature selection, and reference construction

scRNA-seq mixology: towards better benchmarking of single cell RNA-seq analysis methods

Benchmarking single cell RNA-sequencing analysis pipelines using mixture control experiments

Assessment of Machine Learning Methods for Classification in Single Cell ATAC-seq

A multicenter study benchmarking single-cell RNA sequencing technologies using reference samples

Software Benchmark—Classification Tree Algorithms for Cell Atlases Annotation Using Single-Cell RNA-Sequencing Data

A Strategy to Compare Single-Cell RNA Sequencing Data Sets Provides Phenotypic Insight into Cellular Heterogeneity Underlying Biological Similarities and Differences Between Samples

A systematic performance evaluation of clustering methods for single-cell RNA-seq data

Comparison of Single Cell Transcriptome Sequencing Methods: Of Mice and Men

A benchmark of batch-effect correction methods for single-cell RNA sequencing data

A comparison of scRNA-seq annotation methods based on experimentally labeled immune cell subtype dataset

scAnnotatR: framework to accurately classify cell types in single-cell RNA-sequencing data

Comparison of algorithms used in single-cell transcriptomic data analysis

Systematic Comparative Analysis of Single-Nucleotide Variant Detection Methods from Single-Cell RNA Sequencing Data

Benchmarking single-cell RNA-sequencing protocols for cell atlas projects