Abstract:Background Our understanding of the composition, function, and health implications of human microbiota has been advanced by high-throughput sequencing and the development of new genomic analyses. However, trade-offs among alternative strategies for the acquisition and analysis of sequence data remain understudied. Methods We assessed eight popular taxonomic profiling pipelines; MetaPhlAn2, metaMix, PathoScope 2.0, Sigma, Kraken, ConStrains, Centrifuge and Taxator-tk, against a battery of metagenomic datasets simulated from real data. The metagenomic datasets were modeled on 426 complete or permanent draft genomes stored in the Human Oral Microbiome Database and were designed to simulate various experimental conditions, both in the design of a putative experiment; read length (75–1,000 bp reads), sequence depth (100K–10M), and in metagenomic composition; number of species present (10, 100, 426), species distribution. The sensitivity and specificity of each of the pipelines under various scenarios were measured. We also estimated the relative root mean square error and average relative error to assess the abundance estimates produced by different methods. Additional datasets were generated for five of the pipelines to simulate the presence within a metagenome of an unreferenced species, closely related to other referenced species. Additional datasets were also generated in order to measure computational time on datasets of ever-increasing sequencing depth (up to 6 × 10 7 ). Results Testing of eight pipelines against 144 simulated metagenomic datasets initially produced 1,104 discrete results. Pipelines using a marker gene strategy; MetaPhlAn2 and ConStrains, were overall less sensitive, than other pipelines; with the notable exception of Taxator-tk. This difference in sensitivity was largely made up in terms of runtime, significantly lower than more sensitive pipelines that rely on whole-genome alignments such as PathoScope2.0. However, pipelines that used strategies to speed-up alignment between genomic references and metagenomic reads, such as kmerization, were able to combine both high sensitivity and low run time, as is the case with Kraken and Centrifuge. Absent species genomes in the database mostly led to assignment of reads to the most closely related species available in all pipelines. Our results therefore suggest that taxonomic profilers that use kmerization have largely superseded those that use gene markers, coupling low run times with high sensitivity and specificity. Taxonomic profilers using more time-consuming read reassignment, such as PathoScope 2.0, provided the most sensitive profiles under common metagenomic sequencing scenarios. All the results described and discussed in this paper can be visualized using the dedicated R Shiny application ( https://github.com/microgenomics/HumanMicrobiomeAnalysis ). All of our datasets, pipelines and results are made available through the GitHub repository for future benchmarking.

Assessing the performance of current strain resolution tools on long-read metagenomes

An integrated strain-level analytic pipeline utilizing longitudinal metagenomic data

Strainy: phasing and assembly of strain haplotypes from long-read metagenome sequencing

Accurate Reconstruction of Microbial Strains from Metagenomic Sequencing Using Representative Reference Genomes

Scalable Microbial Strain Inference in Metagenomic Data Using StrainFacts

PStrain: an iterative microbial strains profiling algorithm for shotgun metagenomic sequencing data

Strain tracking in complex microbiomes using synteny analysis reveals per-species modes of evolution

StrainSelect: A novel microbiome reference database that disambiguates all bacterial strains, genome assemblies and extant cultures worldwide

Evaluation of computational methods for human microbiome analysis using simulated data

Easing genomic surveillance: A comprehensive performance evaluation of long-read assemblers across multi-strain mixture data of HIV-1 and Other pathogenic viruses for constructing a user-friendly bioinformatic pipeline

Evaluation of metagenomic, 16S rRNA gene and ultra-plexed PCR-based sequencing approaches for profiling antimicrobial resistance gene and bacterial taxonomic composition of polymicrobial samples

Recovery of strain-resolved genomes from human microbiome through an integration framework of single-cell genomics and metagenomics

Analyzing the performance of short-read classification tools on metagenomic samples toward proper diagnosis of diseases

InStrain enables population genomic analysis from metagenomic data and rigorous detection of identical microbial strains

Benchmarking short and long read polishing tools for nanopore assemblies: achieving near-perfect genomes for outbreak isolates

Breaking free from references: a consensus-based approach for community profiling with long amplicon nanopore data

Evaluation of 16S rRNA gene sequencing for species and strain-level microbiome analysis

A comparison of software for analysis of rare and common short tandem repeat (STR) variation using human genome sequences from clinical and population-based samples

Floria: fast and accurate strain haplotyping in metagenomes

Strain/species identification in metagenomes using genome-specific markers

Strain-GeMS: Optimized Subspecies Identification from Microbiome Data Based on Accurate Variant Modeling.