Abstract:An important problem on social information sites is the recovery of ground truth from individual reports when the experts are in the minority. The wisdom of the crowd, i.e. the collective opinion of a group of individuals fails in such a scenario. However, the surprisingly popular (SP) algorithm~\cite{prelec2017solution} can recover the ground truth even when the experts are in the minority, by asking the individuals to report additional prediction reports--their beliefs about the reports of others. Several recent works have extended the surprisingly popular algorithm to an equivalent voting rule (SP-voting) to recover the ground truth ranking over a set of $m$ alternatives. However, we are yet to fully understand when SP-voting can recover the ground truth ranking, and if so, how many samples (votes and predictions) it needs. We answer this question by proposing two rank-order models and analyzing the sample complexity of SP-voting under these models. In particular, we propose concentric mixtures of Mallows and Plackett-Luce models with $G (\ge 2)$ groups. Our models generalize previously proposed concentric mixtures of Mallows models with $2$ groups, and we highlight the importance of $G > 2$ groups by identifying three distinct groups (expert, intermediate, and non-expert) from existing datasets. Next, we provide conditions on the parameters of the underlying models so that SP-voting can recover ground-truth rankings with high probability, and also derive sample complexities under the same. We complement the theoretical results by evaluating SP-voting on simulated and real datasets.

Bayesian Inference for Vertex-Series-Parallel Partial Orders

Learning directed acyclic graphs based on sparsest permutations

Bayesian inferences on uncertain ranks and orderings: Application to ranking players and lineups

Bayesian estimation for longitudinal data in a joint model with HPCs

Belief propagation for permutations, rankings, and partial orders

Posterior propriety in Bayesian extreme value analyses using reference priors

Bayesian Rank-Clustering

Ordinal Regression With Sparse Bayesian

Xorder: A Model Agnostic Post-Processing Framework for Achieving Ranking Fairness While Maintaining Algorithm Utility.

Proximity penalty priors for Bayesian mixture models

Informative extended Mallows priors in the Bayesian Mallows model

Statistical Models of Top-$k$ Partial Orders

A Parallel Algorithm for Exact Bayesian Structure Discovery in Bayesian Networks

Ordering states and channels based on positive Bayesian evidence

Using Pairwise Ordering Preferences to Estimate Causal Effects in Gene Expression from a Mixture of Observational and Intervention Experiments

A new probabilistic model for top-k ranking problem.

Sparse Bayesian Learning for Ranking.

Surprisingly Popular Voting for Concentric Rank-Order Models

Fast Bayesian estimation of spatial count data models

Vertex nomination schemes for membership prediction

Fast post-process Bayesian inference with Variational Sparse Bayesian Quadrature