Abstract:The algorithms required for phylogenetics -- multiple sequence alignment and phylogeny estimation -- are both compute intensive. As the size of DNA sequence datasets continues to increase, there is a need for a tool that can effectively lessen the computational burden associated with this widely used analysis. 'diverse-seq' implements computationally efficient alignment-free algorithms that enable efficient prototyping for phylogenetic workflows. It can accelerate parameter selection searches for sequence alignment and phylogeny estimation by identifying a subset of sequences that are representative of the diversity in a collection. We show that selecting representative sequences with an entropy measure of $k$-mer frequencies correspond well to sampling via conventional genetic distances. The computational performance is linear with respect to the number of sequences and can be run in parallel. Applied to a collection of 10.5k whole microbial genomes on a laptop took ~8 minutes to prepare the data and 4 minutes to select 100 representatives. 'diverse-seq' can further boost the performance of phylogenetic estimation by providing a seed phylogeny that can be further refined by a more sophisticated algorithm. For ~1k whole microbial genomes on a laptop, it takes ~1.8 minutes to estimate a bifurcating tree from mash distances. The 'diverse-seq' algorithms are not limited to homologous sequences. As such, they can improve the performance of other workflows. For instance, machine learning projects that involve non-homologous sequences can benefit as representative sampling can mitigate biases from imbalanced groups. 'diverse-seq' is a BSD-3 licensed Python package that provides both a command-line interface and 'cogent3' plugins. The latter simplifies integration by users into their own analyses. It is available via the Python Package Index and GitHub.

Fast-Part: Fast and Accurate Data Partitioning for Biological Sequence Analysis

A Fast Exact Pattern Matching Algorithm for Biological Sequences

A Fast Improved Pattern Matching Algorithm for Biological Sequences

ProtParts, an automated web server for clustering and partitioning protein datasets

CD-HIT: accelerated for clustering the next-generation sequencing data

diverse-seq: an application for alignment-free selecting and clustering biological sequences

Fast and accurate protein structure search with Foldseek

Pyfastx: a Robust Python Package for Fast Random Access to Sequences from Plain and Gzipped FASTA/Q Files

Fast and sensitive protein alignment using DIAMOND

FastMG: a simple, fast, and accurate maximum likelihood procedure to estimate amino acid replacement rate matrices from large data sets

Ksak: A high-throughput tool for alignment-free phylogenetics

Parallel Read Partitioning for Concurrent Assembly of Metagenomic Data

Fast and accurate short read alignment with hybrid hash-tree data structure

FoldExplorer: Fast and Accurate Protein Structure Search with Sequence-Enhanced Graph Embedding

LinearPartition: linear-time approximation of RNA folding partition function and base-pairing probabilities

SeqKit: A Cross-Platform and Ultrafast Toolkit for FASTA/Q File Manipulation

Real-time Taxonomic Characterization of Long-read Mixed-species Sequencing Samples in Sorted Motif Distance Space:

Sample-Align-D: A High Performance Multiple Sequence Alignment System using Phylogenetic Sampling and Domain Decomposition

A Fast Longest Common Subsequence Algorithm for Biosequences Alignment

A Novel Fast And Memory Efficient Parallel Mlcs Algorithm For Long And Large-Scale Sequences Alignments

SpanSeq: Similarity-based sequence data splitting method for improved development and assessment of deep learning projects