Abstract:Single-cell RNA sequencing (scRNA-seq) is a powerful tool to profile the transcriptomes of a large number of individual cells at a high resolution. These data usually contain measurements of gene expression for many genes in thousands or tens of thousands of cells, though some datasets now reach the million-cell mark. Projecting high-dimensional scRNA-seq data into a low dimensional space aids downstream analysis and data visualization. Many recent preprints accomplish this using variational autoencoders (VAE), generative models that learn underlying structure of data by compress it into a constrained, low dimensional space. The low dimensional spaces generated by VAEs have revealed complex patterns and novel biological signals from large-scale gene expression data and drug response predictions. Here, we evaluate a simple VAE approach for gene expression data, Tybalt, by training and measuring its performance on sets of simulated scRNA-seq data. We find a number of counter-intuitive performance features: i.e., deeper neural networks can struggle when datasets contain more observations under some parameter configurations. We show that these methods are highly sensitive to parameter tuning: when tuned, the performance of the Tybalt model, which was not optimized for scRNA-seq data, outperforms other popular dimension reduction approaches - PCA, ZIFA, UMAP and t-SNE. On the other hand, without tuning performance can also be remarkably poor on the same data. Our results should discourage authors and reviewers from relying on self-reported performance comparisons to evaluate the relative value of contributions in this area at this time. Instead, we recommend that attempts to compare or benchmark autoencoder methods for scRNA-seq data be performed by disinterested third parties or by methods developers only on unseen benchmark data that are provided to all participants simultaneously because the potential for performance differences due to unequal parameter tuning is so high.

Tuning hyperparameters of doublet-detection methods for single-cell RNA sequencing data

DoubletFinder: Doublet Detection in Single-Cell RNA Sequencing Data Using Artificial Nearest Neighbors

Protocol for Executing and Benchmarking Eight Computational Doublet-Detection Methods in Single-Cell RNA Sequencing Data Analysis

Doublet identification in single-cell sequencing data using scDblFinder

scIBD: a self-supervised iterative-optimizing model for boosting the detection of heterotypic doublets in single-cell chromatin accessibility data

Optimal distance metrics for single-cell RNA-seq populations

Demuxafy: improvement in droplet assignment by integrating multiple single-cell demultiplexing and doublet detection methods

Chord: an ensemble machine learning algorithm to identify doublets in single-cell RNA sequencing data

Vaeda computationally annotates doublets in single-cell RNA sequencing data

Beyond benchmarking and towards predictive models of dataset-specific single-cell RNA-seq pipeline performance

More cells, more doublets in highly multiplexed single-cell data

Beyond benchmarking: towards predictive models of dataset-specific single-cell RNA-seq pipeline performance

Benchmark and Parameter Sensitivity Analysis of Single-Cell RNA Sequencing Clustering Methods

Parameter tuning is a key part of dimensionality reduction via deep variational autoencoders for single cell RNA transcriptomics

An optimized protocol for single cell transcriptional profiling by combinatorial indexing

Trade-off between conservation of biological variation and batch effect removal in deep generative modeling for single-cell transcriptomics

Optimized single-nucleus transcriptional profiling by combinatorial indexing

Optimization and Redevelopment of Single-Cell Data Analysis Workflow Based on Deep Generative Models

scRNA-seq mixology: towards better benchmarking of single cell RNA-seq analysis methods

Accounting for technical noise in differential expression analysis of single-cell RNA sequencing data