Abstract:The metagenome analysis of complex environments with thousands of datasets, such as those available in the Sequence Read Archive, requires immense computational resources to complete the computational work within an acceptable time frame. Such large-scale analyses require that the underlying infrastructure is used efficiently. In addition, any analysis should be fully reproducible and the workflow must be publicly available to allow other researchers to understand the reasoning behind computed results. Here, we introduce the Metagenomics-Toolkit, a scalable, data agnostic workflow that automates the analysis of short and long metagenomic reads obtained from Illumina or Oxford Nanopore Technology devices, respectively. The Metagenomics-Toolkit offers not only standard features expected in a metagenome workflow, such as quality control, assembly, binning, and annotation, but also distinctive features, such as plasmid identification based on various tools, the recovery of unassembled microbial community members and the discovery of microbial interdependencies through a combination of dereplication, co-occurrence, and genome-scale metabolic modeling. Furthermore, the Metagenomics-Toolkit includes a machine learning-optimized assembly step that tailors the peak RAM value requested by a metagenome assembler to match actual requirements, thereby minimizing the dependency on dedicated high-memory hardware. While the Metagenomics Toolkit can be executed on user workstations, it also offers several optimizations for an efficient cloud-based cluster execution. We compare the Metagenomics-Toolkit to five commonly used metagenomics workflows and demonstrate the capabilities of the Metagenomics-Toolkit by executing it on 757 metagenome datasets from sewage samples for an investigation of a possible sewage core microbiome. The Metagenomics-Toolkit is open source and available at https://github.com/metagenomics/metagenomics-tk.

CELEBRIMBOR: core and accessory genes from metagenomes

RIBAP: a comprehensive bacterial core genome annotation pipeline for pangenome calculation beyond the species level

Panacus: fast and exact pangenome growth and core size estimation

Pandagma: a tool for identifying pan-gene sets and gene families at desired evolutionary depths and accommodating whole-genome duplications

Toward an efficient method of identifying core genes for evolutionary and functional microbial phylogenies

Towards complete representation of bacterial contents in metagenomic samples

Estimation of genomic characteristics by analyzing k-mer frequency in de novo genome projects

MetaCerberus: distributed highly parallelized HMM-based processing for robust functional annotation across the tree of life

Large-scale estimation of bacterial and archaeal DNA prevalence in metagenomes reveals biome-specific patterns

PanGeneHome : A Web Interface to Analyze Microbial Pangenomes

BusyBee Web: metagenomic data analysis by bootstrapped supervised binning and annotation

RefSeq and the prokaryotic genome annotation pipeline in the age of metagenomes

Accurate Genome Relative Abundance Estimation Based on Shotgun Metagenomic Reads

VBCG: 20 validated bacterial core genes for phylogenomic analysis with high fidelity and resolution

Metagenomics-Toolkit: The Flexible and Efficient Cloud-Based Metagenomics Workflow featuring Machine Learning-Enabled Resource Allocation

BUSCO: assessing genome assembly and annotation completeness with single-copy orthologs

Disentangle genus microdiversity within a complex microbial community by using a multi-distance long-read binning method: example of Candidatus Accumulibacter

mmgenome: a toolbox for reproducible genome extraction from metagenomes

MarkerScan: Separation and assembly of cobionts sequenced alongside target species in biodiversity genomics projects

Exploring environmental intra‐species diversity through non‐redundant pangenome assemblies

Efficient High-Quality Metagenome Assembly from Long Accurate Reads using Minimizer-space de Bruijn Graphs