Abstract:The Neighbor-Joining (NJ) algorithm is a widely used method for constructing phylogenetic trees from genetic distances. While NJ is known to perform well with tree-like data, its behavior under admixture remains understudied. In this work, we present a geometric framework for analyzing the NJ algorithm under a linear admixture model. We focus on three key properties related to clustering order, distance, and topological path length in the resulting NJ trees involving five taxa. Our approach leverages polyhedral geometry to define NJ cones, which correspond to distinct cherry-picking orders and partition the space of dissimilarity vectors. We project dissimilarity vectors with admixture into a lower-dimensional space without admixture, defining polyhedral regions induced by NJ cones that satisfy specified properties. We compute the exact probabilities that these properties hold by directly calculating the volumes of the induced NJ cones and compare them with Monte Carlo integration and standard NJ simulation methods. Our results show that the property on clustering order is always satisfied, while the other properties are highly probable but depend on the admixture fraction. We also prove that certain induced NJ cones have zero volume, indicating that the corresponding NJ tree topologies are infeasible under admixture. We have implemented our methods as a publicly available NeighborJoining within Macaulay2, providing an efficient tool for analyzing NJ cones and their properties. This work provides new insights into the geometric structure inherent to the NJ algorithm in the presence of admixture, identifying the conditions under which admixture influences the resulting phylogenetic trees.

Neighbor Joining And Leaf Status

On the optimality of the neighbor-joining algorithm

Combinatorial and computational investigations of Neighbor-Joining bias

The Neighbor-Net Algorithm

The Geometry of the Neighbor-Joining Algorithm for Small Trees

Species tree estimation using Neighbor Joining

Admixed populations in the neighbor-joining algorithm: a geometric analysis with five taxa

Random Local Neighbor Joining: a New Method for Reconstructing Phylogenetic Trees.

The neighbor-joining method: a new method for reconstructing phylogenetic trees.

Optimality of the Neighbor Joining Algorithm and Faces of the Balanced Minimum Evolution Polytope

Stochastic safety radius on Neighbor-Joining method and Balanced Minimal Evolution on small trees

Recrafting the neighbor-joining method

Efficiency of the Neighbor-Joining Method in Reconstructing Deep and Shallow Evolutionary Relationships in Large Phylogenies

Scaling neighbor joining to one million taxa with dynamic and heuristic neighbor joining

Relaxed Neighbor Joining: A Fast Distance-Based Phylogenetic Tree Construction Method

THE ACCURACY OF PHYLOGENETIC ESTIMATION USING THE NEIGHBOR‐JOINING METHOD

A topological transformation in evolutionary tree search methods based on maximum likelihood combining p-ECR and neighbor joining

Phylogenies in yeast species descriptions: in defense of neighbor‐joining

A 'stochastic safety radius' for distance-based tree reconstruction

Species, Clusters and the 'Tree of Life': A graph-theoretic perspective

From tree matching to sparse graph alignment