Five-leaf generalizations of the D-statistic reveal the directionality of admixture
Kalle Leppälä,Flavio Augusto da Silva Coelho,Michaela Richter,Victor A Albert,Charlotte Lindqvist
DOI: https://doi.org/10.1101/2024.02.24.581856
2024-09-02
Abstract:Over the past 15 years, the D-statistic, a four-taxon test for organismal admixture (hybridization, or introgression) that incorporates single nucleotide polymorphism data with allelic patterns ABBA and BABA, has seen considerable use. This statistic seeks to discern significant deviation from either a given species tree assumption, or from the balanced incomplete lineage sorting that could otherwise defy this species tree. However, while the D-statistic can successfully discriminate admixture from incomplete lineage sorting, it is not a simple matter to determine the directionality of admixture using only four-leaf tree models. As such, methods have been developed that use 5 leaves to evaluate admixture trajectories. Among these, the DFOIL method, which tests allelic patterns on the "symmetric" tree S = (((1, 2),(3, 4)), 5), succeeds in finding admixture direction for many five-taxon examples. However, DFOIL does not make full use of all symmetry, nor can DFOIL properly accommodate the use of singleton patterns (such as BAAAA and ABAAA) when ancient samples are included. The latter becomes problematic since "mutational time" elapsed since ancient organism death is unequal to that of modern samples, and because damaged DNA can bias results in unknown ways. Here, we take inspiration from DFOIL to develop a new and completely general family of five-leaf admixture tests, dubbed Δ-statistics, that can either incorporate or exclude the singleton allelic patterns depending on individual taxon and age sampling choices. We describe two new shapes that are also fully testable, namely the "asymmetric" tree A = ((((1, 2), 3), 4), 5) and the "quasisymmetric" tree Q = (((1, 2), 3),(4, 5)), that can considerably supplement the "symmetric" S = (((1, 2),(3, 4)), 5) model used by DFOIL. We demonstrate the consistency of Δ-statistics under various simulated scenarios, and provide empirical examples using data from black, brown and polar bears, the latter also including two ancient samples from previous studies. Recently DFOIL and one of these ancient samples was used to argue for a dominant polar bear → brown bear introgression direction. However, we find, using both this ancient polar bear and our own, that by far the strongest signal using both DFOIL and Δ-statistics on tree S is actually bidirectional gene flow of indistinguishable direction. Further experiments on trees A and Q instead highlight what were likely two phases of admixture: one with stronger brown bear → polar bear introgression in ancient times, and a recent phase with predominant polar bear → brown bear directionality. Code and documentation available at https://github.com/KalleLeppala/Delta-statistics
Genetics