Abstract:The Neighbor-Joining (NJ) algorithm is a widely used method for constructing phylogenetic trees from genetic distances. While NJ is known to perform well with tree-like data, its behavior under admixture remains understudied. In this work, we present a geometric framework for analyzing the NJ algorithm under a linear admixture model. We focus on three key properties related to clustering order, distance, and topological path length in the resulting NJ trees involving five taxa. Our approach leverages polyhedral geometry to define NJ cones, which correspond to distinct cherry-picking orders and partition the space of dissimilarity vectors. We project dissimilarity vectors with admixture into a lower-dimensional space without admixture, defining polyhedral regions induced by NJ cones that satisfy specified properties. We compute the exact probabilities that these properties hold by directly calculating the volumes of the induced NJ cones and compare them with Monte Carlo integration and standard NJ simulation methods. Our results show that the property on clustering order is always satisfied, while the other properties are highly probable but depend on the admixture fraction. We also prove that certain induced NJ cones have zero volume, indicating that the corresponding NJ tree topologies are infeasible under admixture. We have implemented our methods as a publicly available NeighborJoining within Macaulay2, providing an efficient tool for analyzing NJ cones and their properties. This work provides new insights into the geometric structure inherent to the NJ algorithm in the presence of admixture, identifying the conditions under which admixture influences the resulting phylogenetic trees.

The neighbor-joining method: a new method for reconstructing phylogenetic trees.

Random Local Neighbor Joining: a New Method for Reconstructing Phylogenetic Trees.

Recrafting the neighbor-joining method

Relaxed Neighbor Joining: A Fast Distance-Based Phylogenetic Tree Construction Method

Species tree estimation using Neighbor Joining

THE ACCURACY OF PHYLOGENETIC ESTIMATION USING THE NEIGHBOR‐JOINING METHOD

QUARTET CONSISTENCY COUNT METHOD FOR RECONSTRUCTING PHYLOGENETIC TREES

Distance-based phylogenetic methods around a polytomy

The Neighbor-Net Algorithm

The Geometry of the Neighbor-Joining Algorithm for Small Trees

Combinatorial and computational investigations of Neighbor-Joining bias

Efficiency of the Neighbor-Joining Method in Reconstructing Deep and Shallow Evolutionary Relationships in Large Phylogenies

On the optimality of the neighbor-joining algorithm

Estimating Species Trees from Quartet Gene Tree Distributions under the Coalescent Model

Stochastic safety radius on Neighbor-Joining method and Balanced Minimal Evolution on small trees

Admixed populations in the neighbor-joining algorithm: a geometric analysis with five taxa

Scaling neighbor joining to one million taxa with dynamic and heuristic neighbor joining

A Fast and Scalable Method for Inferring Phylogenetic Networks from Trees by Aligning Lineage Taxon Strings

Scalable distance-based phylogeny inference using divide-and-conquer

TreeFormer: A transformer-based tree rearrangement operation for phylogenetic reconstruction

Statistical Phylogenetic Tree Analysis Using Differences of Means