Abstract:BACKGROUND:There are a variety of bioinformatic pipelines and downstream analysis methods for analyzing 16S rRNA marker-gene surveys. However, appropriate assessment datasets and metrics are needed as there is limited guidance to decide between available analysis methods. Mixtures of environmental samples are useful for assessing analysis methods as one can evaluate methods based on calculated expected values using unmixed sample measurements and the mixture design. Previous studies have used mixtures of environmental samples to assess other sequencing methods such as RNAseq. But no studies have used mixtures of environmental to assess 16S rRNA sequencing.RESULTS:We developed a framework for assessing 16S rRNA sequencing analysis methods which utilizes a novel two-sample titration mixture dataset and metrics to evaluate qualitative and quantitative characteristics of count tables. Our qualitative assessment evaluates feature presence/absence exploiting features only present in unmixed samples or titrations by testing if random sampling can account for their observed relative abundance. Our quantitative assessment evaluates feature relative and differential abundance by comparing observed and expected values. We demonstrated the framework by evaluating count tables generated with three commonly used bioinformatic pipelines: (i) DADA2 a sequence inference method, (ii) Mothur a de novo clustering method, and (iii) QIIME an open-reference clustering method. The qualitative assessment results indicated that the majority of Mothur and QIIME features only present in unmixed samples or titrations were accounted for by random sampling alone, but this was not the case for DADA2 features. Combined with count table sparsity (proportion of zero-valued cells in a count table), these results indicate DADA2 has a higher false-negative rate whereas Mothur and QIIME have higher false-positive rates. The quantitative assessment results indicated the observed relative abundance and differential abundance values were consistent with expected values for all three pipelines.CONCLUSIONS:We developed a novel framework for assessing 16S rRNA marker-gene survey methods and demonstrated the framework by evaluating count tables generated with three bioinformatic pipelines. This framework is a valuable community resource for assessing 16S rRNA marker-gene survey bioinformatic methods and will help scientists identify appropriate analysis methods for their marker-gene surveys.

Evaluating long-read RNA-sequencing analysis tools with in silico mixtures

Benchmarking long-read RNA-sequencing analysis tools using in silico mixtures

Study of Quality Assessment of Ion Sphere Particles for Ultra-high Throughput Sequencing

Assembly Arena: Benchmarking RNA isoform reconstruction algorithms for nanopore sequencing

scRNA-seq mixology: towards better benchmarking of single cell RNA-seq analysis methods

Benchmarking single cell RNA-sequencing analysis pipelines using mixture control experiments

A Framework for Assessing 16S Rrna Marker-Gene Survey Data Analysis Methods Using Mixtures.

Comprehensive Assessment of Isoform Detection Methods for Third-Generation Sequencing Data

Comprehensive evaluation of RNA-seq alignment methods based on long-read sequencing data.

Comprehensive assessment of mRNA isoform detection methods for long-read sequencing data

The shaky foundations of simulating single-cell RNA sequencing data

Contrasting and combining transcriptome complexity captured by short and long RNA sequencing reads

Enhancing novel isoform discovery: leveraging nanopore long-read sequencing and machine learning approaches

Assessing 16S marker gene survey data analysis methods using mixtures of human stool sample DNA extracts.

A Real-World Multi-Center RNA-seq Benchmarking Study Using the Quartet and MAQC Reference Materials

A Comprehensive Evaluation of Alignment Software for Reduced Representation Bisulfite Sequencing Data

A comprehensive benchmarking of differential splicing tools for RNA-seq analysis at the event level

Enhancing RNA-seq analysis by addressing all co-existing biases using a self-benchmarking approach with 2D structural insights

Evaluation and comparison of computational tools for RNA-seq isoform quantification

InSilicoSeq 2.0: Simulating realistic amplicon-based sequence reads

Benchmarking of bioinformatics tools for the hybrid de novo assembly of human whole-genome sequencing data