The coalescent in finite populations with arbitrary, fixed structure

Benjamin Allen,Alex McAvoy

DOI: https://doi.org/10.1016/j.tpb.2024.06.004

2024-06-29

Abstract:The coalescent is a stochastic process representing ancestral lineages in a population undergoing neutral genetic drift. Originally defined for a well-mixed population, the coalescent has been adapted in various ways to accommodate spatial, age, and class structure, along with other features of real-world populations. To further extend the range of population structures to which coalescent theory applies, we formulate a coalescent process for a broad class of neutral drift models with arbitrary -- but fixed -- spatial, age, sex, and class structure, haploid or diploid genetics, and any fixed mating pattern. Here, the coalescent is represented as a random sequence of mappings $\mathcal{C} = \left(C_t\right)_{t=0}^\infty$ from a finite set $G$ to itself. The set $G$ represents the ``sites'' (in individuals, in particular locations and/or classes) at which these alleles can live. The state of the coalescent, $C_t:G \to G$, maps each site $g \in G$ to the site containing $g$'s ancestor, $t$ time-steps into the past. Using this representation, we define and analyze coalescence time, coalescence branch length, mutations prior to coalescence, and stationary probabilities of identity-by-descent and identity-by-state. For low mutation, we provide a recipe for computing identity-by-descent and identity-by-state probabilities via the coalescent. Applying our results to a diploid population with arbitrary sex ratio $r$, we find that measures of genetic dissimilarity, among any set of sites, are scaled by $4r(1-r)$ relative to the even sex ratio case.

Populations and Evolution,Probability

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to extend and refine the existing coalescent theory so that it can be applied to the neutral genetic drift model in finite populations with arbitrary but fixed structures. Specifically, the authors aim to:

1. **Construct a general framework**: Develop a coalescent process applicable to a wide range of neutral drift models, which can include arbitrary spatial, age, gender, class structures, as well as fixed mating patterns. This enables the coalescent theory to be more widely applied to actual populations, rather than just idealized, well - mixed populations.

2. **Analyze key genetic quantities**: Define and analyze the coalescent time, the length of the coalescent branch, the number of mutations before coalescence, and the steady - state probabilities of homology and homoplasy within this framework. These quantities are crucial for understanding the genetic diversity within a population and its evolutionary dynamics.

3. **Handle low - mutation cases**: Provide a method for calculating the probabilities of homology and homoplasy under conditions of low mutation rates. This is of great significance for studying the evolution of genetic diversity and social behavior in actual populations, especially in the case of non - linear and multi - party interactions.

4. **Connect classical theory with new models**: Demonstrate through specific cases how to recover classical coalescent results from their formal methods, thereby providing new tools and perspectives for research in theoretical genetics and evolutionary biology.

The core contribution of the paper lies in its provision of a mathematically rigorous and flexible framework that can handle the problem of neutral genetic drift in finite populations with complex structures, thus providing new research means for multiple fields in genetics and evolutionary biology.