Abstract:The Bradley-Terry-Luce (BTL) model is one of the most widely used models for ranking a collection of items or agents based on pairwise comparisons among them. Given $n$ agents, the BTL model endows each agent $i$ with a latent skill score $\alpha_i > 0$ and posits that the probability that agent $i$ is preferred over agent $j$ is $\alpha_i/(\alpha_i + \alpha_j)$. In this work, our objective is to formulate a hypothesis test that determines whether a given pairwise comparison dataset, with $k$ comparisons per pair of agents, originates from an underlying BTL model. We formalize this testing problem in the minimax sense and define the critical threshold of the problem. We then establish upper bounds on the critical threshold for general induced observation graphs (satisfying mild assumptions) and develop lower bounds for complete induced graphs. Our bounds demonstrate that for complete induced graphs, the critical threshold scales as $\Theta((nk)^{-1/2})$ in a minimax sense. In particular, our test statistic for the upper bounds is based on a new approximation we derive for the separation distance between general pairwise comparison models and the class of BTL models. To further assess the performance of our statistical test, we prove upper bounds on the type I and type II probabilities of error. Much of our analysis is conducted within the context of a fixed observation graph structure, where the graph possesses certain ``nice'' properties, such as expansion and bounded principal ratio. Additionally, we derive several auxiliary results, such as bounds on principal ratios of graphs, $\ell^2$-bounds on BTL parameter estimation under model mismatch, stability of rankings under the BTL model, etc. We validate our theoretical results through experiments on synthetic and real-world datasets and propose a data-driven permutation testing approach to determine test thresholds.

On Extending the Bradley-Terry Model to Accommodate Ties in Paired Comparison Experiments

Ties in Paired-Comparison Experiments: A Generalization of the Bradley-Terry Model

The many routes to the ubiquitous Bradley-Terry model

Models for Paired Comparison Data: A Review with Emphasis on Dependent Data

Reward Learning From Preference With Ties

A note on extending Scheffé's modified multiple-comparison procedure to other analysis situations

The incomplete Analytic Hierarchy Process and Bradley-Terry model: (in)consistency and information retrieval

Minimax Hypothesis Testing for the Bradley-Terry-Luce Model

The Bradly-Terry Regression Trunk approach for modelling preference data with small trees

On Extending Direct Preference Optimization to Accommodate Ties

Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives

Statistical inference for pairwise comparison models

A Bayesian Bradley-Terry model to compare multiple ML algorithms on multiple data sets

The Bradley-Terry Regression Trunk approach for Modeling Preference Data with Small Trees

Comparative analysis of the existence and uniqueness conditions of parameter estimation in paired comparison models

Bayesian analysis of paired-comparison sound quality ratings

Ranking Inferences Based on the Top Choice of Multiway Comparisons

Experimental Design under the Bradley-Terry Model

A Bayesian approach to time-varying latent strengths in pairwise comparisons

Optimal Designs for Second-Order Interactions in Paired Comparison Experiments with Binary Attributes

Hierarchical Bayesian Bradley-Terry for Applications in Major League Baseball