Abstract:The field of population genetics attempts to advance our understanding of evolutionary processes. It has applications, for example, in medical research, wildlife conservation, and - in conjunction with recent advances in ancient DNA sequencing technology - studying human migration patterns over the past few thousand years. The basic toolbox of population genetics includes genealogical tress, which describe the shared evolutionary history among individuals of the same species. They are calculated on the basis of genetic variations. However, in recombining organisms, a single tree is insufficient to describe the evolutionary history of the whole genome. Instead, a collection of correlated trees can be used, where each describes the evolutionary history of a consecutive region of the genome. The current corresponding state of-the-art data structure, tree sequences, compresses these genealogical trees via edit operations when moving from one tree to the next along the genome instead of storing the full, often redundant, description for each tree. We propose a new data structure, genealogical forests, which compresses the set of genealogical trees into a DAG. In this DAG identical subtrees that are shared across the input trees are encoded only once, thereby allowing for straight-forward memoization of intermediate results. Additionally, we provide a C++ implementation of our proposed data structure, called gfkit, which is 2.1 to 11.2 (median 4.0) times faster than the state-of-the-art tool on empirical and simulated datasets at computing important population genetics statistics such as the Allele Frequency Spectrum, Patterson's f, the Fixation Index, Tajima's D, pairwise Lowest Common Ancestors, and others. On Lowest Common Ancestor queries with more than two samples as input, gfkit scales asymptotically better than the state-of-the-art, and is thus up to 990 times faster. In conclusion, our proposed data structure compresses genealogical trees by storing shared subtrees only once, thereby enabling straight-forward memoization of intermediate results, yielding a substantial runtime reduction and a potentially more intuitive data representation over the state-of-the-art. Our improvements will boost the development of novel analyses and models in the field of population genetics and increases scalability to ever-growing genomic datasets.

Genetic algorithm approaches for the phylogenetic analysis of large biological sequence datasets under the maximum likelihood criterion

ParGenes: a tool for massively parallel model selection and phylogenetic tree inference on thousands of genes

Many-core algorithms for high-dimensional gradients on phylogenetic trees

IQ-TREE: A Fast and Effective Stochastic Algorithm for Estimating Maximum-Likelihood Phylogenies

RAxML-NG: a fast, scalable and user-friendly tool for maximum likelihood phylogenetic inference

Gradients do grow on trees: a linear-time ${\cal O}\hspace{-0.2em}\left( N \right)$-dimensional gradient for statistical phylogenetics

Efficient Bayesian Inference of General Gaussian Models on Large Phylogenetic Trees

The Influence of the Number of Tree Searches on Maximum Likelihood Inference in Phylogenomics.

PhyloAcc-GT: A Bayesian method for inferring patterns of substitution rate shifts on targeted lineages accounting for gene tree discordance

Harnessing machine learning to guide phylogenetic-tree search algorithms

Fast phylogeny reconstruction through learning of ancestral sequences

OrthoPhyl – Streamlining large scale, orthology-based phylogenomic studies of bacteria at broad evolutionary scales

Phylogenomics with Paralogs

Scalable distance-based phylogeny inference using divide-and-conquer

From Easy to Hopeless-Predicting the Difficulty of Phylogenetic Analyses

Co-Phylog: an Assembly-Free Phylogenomic Approach for Closely Related Organisms

Analysis of Phylogeny Tracking Algorithms for Serial and Multiprocess Applications

Memoization on Shared Subtrees Accelerates Computations on Genealogical Forests

diverse-seq: an application for alignment-free selecting and clustering biological sequences

Computing the probability of gene trees concordant with the species tree in the multispecies coalescent