Abstract:The field of population genetics attempts to advance our understanding of evolutionary processes. It has applications, for example, in medical research, wildlife conservation, and - in conjunction with recent advances in ancient DNA sequencing technology - studying human migration patterns over the past few thousand years. The basic toolbox of population genetics includes genealogical tress, which describe the shared evolutionary history among individuals of the same species. They are calculated on the basis of genetic variations. However, in recombining organisms, a single tree is insufficient to describe the evolutionary history of the whole genome. Instead, a collection of correlated trees can be used, where each describes the evolutionary history of a consecutive region of the genome. The current corresponding state of-the-art data structure, tree sequences, compresses these genealogical trees via edit operations when moving from one tree to the next along the genome instead of storing the full, often redundant, description for each tree. We propose a new data structure, genealogical forests, which compresses the set of genealogical trees into a DAG. In this DAG identical subtrees that are shared across the input trees are encoded only once, thereby allowing for straight-forward memoization of intermediate results. Additionally, we provide a C++ implementation of our proposed data structure, called gfkit, which is 2.1 to 11.2 (median 4.0) times faster than the state-of-the-art tool on empirical and simulated datasets at computing important population genetics statistics such as the Allele Frequency Spectrum, Patterson's f, the Fixation Index, Tajima's D, pairwise Lowest Common Ancestors, and others. On Lowest Common Ancestor queries with more than two samples as input, gfkit scales asymptotically better than the state-of-the-art, and is thus up to 990 times faster. In conclusion, our proposed data structure compresses genealogical trees by storing shared subtrees only once, thereby enabling straight-forward memoization of intermediate results, yielding a substantial runtime reduction and a potentially more intuitive data representation over the state-of-the-art. Our improvements will boost the development of novel analyses and models in the field of population genetics and increases scalability to ever-growing genomic datasets.

CompactTree: A lightweight header-only C++ library for ultra-large phylogenetics

Efficient comparative phylogenetics on large trees

Efficient phylogenetic tree inference for massive taxonomic datasets: harnessing the power of a server to analyze 1 million taxa

DecentTree: scalable Neighbour-Joining for the genomic era

A fast technique for constructing evolutionary tree with the application of compact sets

FastTree: Computing Large Minimum Evolution Trees with Profiles instead of a Distance Matrix

Living Trees: High-Quality Reproducible and Reusable Construction of Bacterial Phylogenetic Trees

Phylo2Vec: a vector representation for binary trees

GrapeTree: Visualization of core genomic relationships among 100,000 bacterial pathogens

EasyCGTree: a pipeline for prokaryotic phylogenomic analysis based on core gene sets

MIKE: an ultrafast, assembly- and alignment-free approach for phylogenetic tree construction

DLTree: Efficient and Accurate Phylogeny Reconstruction Using the Dynamical Language Method

Chronumental: time tree estimation from very large phylogenies

PhyloJS: Bridging phylogenetics and web development with a JavaScript utility library

SaAlign: Multiple DNA/RNA Sequence Alignment and Phylogenetic Tree Construction Tool for Ultra-Large Datasets and Ultra-Long Sequences Based on Suffix Array

Robust expansion of phylogeny for fast-growing genome sequence data

Genomic data analysis in tree spaces

Phylogenetic tree distance computation over succinct representations

Memoization on Shared Subtrees Accelerates Computations on Genealogical Forests

Scalable distance-based phylogeny inference using divide-and-conquer