Abstract:Background Our understanding of the composition, function, and health implications of human microbiota has been advanced by high-throughput sequencing and the development of new genomic analyses. However, trade-offs among alternative strategies for the acquisition and analysis of sequence data remain understudied. Methods We assessed eight popular taxonomic profiling pipelines; MetaPhlAn2, metaMix, PathoScope 2.0, Sigma, Kraken, ConStrains, Centrifuge and Taxator-tk, against a battery of metagenomic datasets simulated from real data. The metagenomic datasets were modeled on 426 complete or permanent draft genomes stored in the Human Oral Microbiome Database and were designed to simulate various experimental conditions, both in the design of a putative experiment; read length (75–1,000 bp reads), sequence depth (100K–10M), and in metagenomic composition; number of species present (10, 100, 426), species distribution. The sensitivity and specificity of each of the pipelines under various scenarios were measured. We also estimated the relative root mean square error and average relative error to assess the abundance estimates produced by different methods. Additional datasets were generated for five of the pipelines to simulate the presence within a metagenome of an unreferenced species, closely related to other referenced species. Additional datasets were also generated in order to measure computational time on datasets of ever-increasing sequencing depth (up to 6 × 10 7 ). Results Testing of eight pipelines against 144 simulated metagenomic datasets initially produced 1,104 discrete results. Pipelines using a marker gene strategy; MetaPhlAn2 and ConStrains, were overall less sensitive, than other pipelines; with the notable exception of Taxator-tk. This difference in sensitivity was largely made up in terms of runtime, significantly lower than more sensitive pipelines that rely on whole-genome alignments such as PathoScope2.0. However, pipelines that used strategies to speed-up alignment between genomic references and metagenomic reads, such as kmerization, were able to combine both high sensitivity and low run time, as is the case with Kraken and Centrifuge. Absent species genomes in the database mostly led to assignment of reads to the most closely related species available in all pipelines. Our results therefore suggest that taxonomic profilers that use kmerization have largely superseded those that use gene markers, coupling low run times with high sensitivity and specificity. Taxonomic profilers using more time-consuming read reassignment, such as PathoScope 2.0, provided the most sensitive profiles under common metagenomic sequencing scenarios. All the results described and discussed in this paper can be visualized using the dedicated R Shiny application ( https://github.com/microgenomics/HumanMicrobiomeAnalysis ). All of our datasets, pipelines and results are made available through the GitHub repository for future benchmarking.

In Silico and Experimental Evaluation of Primer Sets for Species-Level Resolution of the Vaginal Microbiota Using 16S Ribosomal RNA Gene Sequencing

Comparison of Microbial Diversity Determined with the Same Variable Tag Sequence Extracted from Two Different PCR Amplicons

Species-level resolution for the vaginal microbiota with short amplicons

Primer, Pipelines, Parameters: Issues in 16S rRNA Gene Sequencing

Evaluation of 16S rRNA gene sequencing for species and strain-level microbiome analysis

A multi-amplicon 16S rRNA sequencing and analysis method for improved taxonomic profiling of bacterial communities

An optimized 16S rRNA sequencing protocol for vaginal microbiome to avoid biased abundance estimation

Evaluation of general 16S ribosomal RNA gene PCR primers for classical and next-generation sequencing-based diversity studies

Using short-read 16S rRNA sequencing of multiple variable regions to generate high-quality results to a species level.

The Impact of Primer Design on Amplicon-Based Metagenomic Profiling Accuracy: Detailed Insights into Bifidobacterial Community Structure

16S rRNA gene sequencing of mock microbial populations- impact of DNA extraction method, primer choice and sequencing platform

SpeciateIT and vSpeciateDB: Novel, fast and accurate per sequence 16S rRNA gene taxonomic classification of vaginal microbiota

SpeciateIT and vSpeciateDB: novel, fast, and accurate per sequence 16S rRNA gene taxonomic classification of vaginal microbiota

Optimisation of methods for bacterial skin microbiome investigation: primer selection and comparison of the 454 versus MiSeq platform

Multi-amplicon microbiome data analysis pipelines for mixed orientation sequences using QIIME2: Assessing reference database, variable region and pre-processing bias in classification of mock bacterial community samples

Metagenomic profiling pipelines improve taxonomic classification for 16S amplicon sequencing data

An accurate and efficient experimental approach for characterization of the complex oral microbiota

High-resolution microbial community reconstruction by integrating short reads from multiple 16S rRNA regions

Evaluation of computational methods for human microbiome analysis using simulated data

A comparison between Greengenes, SILVA, RDP, and NCBI reference databases in four published microbiota datasets

Short‐ and long‐read metabarcoding of the eukaryotic rRNA operon: evaluation of primers and comparison to shotgun metagenomics sequencing