Abstract:Peer review assignment algorithms aim to match research papers to suitable expert reviewers, working to maximize the quality of the resulting reviews. A key challenge in designing effective assignment policies is evaluating how changes to the assignment algorithm map to changes in review quality. In this work, we leverage recently proposed policies that introduce randomness in peer-review assignment--in order to mitigate fraud--as a valuable opportunity to evaluate counterfactual assignment policies. Specifically, we exploit how such randomized assignments provide a positive probability of observing the reviews of many assignment policies of interest. To address challenges in applying standard off-policy evaluation methods, such as violations of positivity, we introduce novel methods for partial identification based on monotonicity and Lipschitz smoothness assumptions for the mapping between reviewer-paper covariates and outcomes. We apply our methods to peer-review data from two computer science venues: the TPDP'21 workshop (95 papers and 35 reviewers) and the AAAI'22 conference (8,450 papers and 3,145 reviewers). We consider estimates of (i) the effect on review quality when changing weights in the assignment algorithm, e.g., weighting reviewers' bids vs. textual similarity (between the review's past papers and the submission), and (ii) the "cost of randomization", capturing the difference in expected quality between the perturbed and unperturbed optimal match. We find that placing higher weight on text similarity results in higher review quality and that introducing randomization in the reviewer-paper assignment only marginally reduces the review quality. Our methods for partial identification may be of independent interest, while our off-policy approach can likely find use evaluating a broad class of algorithmic matching systems.

Assignment Procedure Biases In Randomised Policy Experiments

Improved Policy Evaluation for Randomized Trials of Algorithmic Resource Allocation

Counterfactual Evaluation of Peer-Review Assignment Policies

Randomization Bias in Field Trials to Evaluate Targeting Methods

Randomization to Randomization Probability: Estimating Treatment Effects under Actual Conditions of Use.

Policy equipoise and interventional superiority

The Multi-Unit Assignment Problem: Theory and Evidence from Course Allocation at Harvard

Disentangling Treatment and Placebo Effects in Randomized Experiments Using Principal Stratification-An Introduction

Randomized experiments to detect and estimate social influence in networks

Uncovering Bias in Order Assignment

Leveraging Random Assignment to Impute Missing Covariates in Causal Studies

Precise unbiased estimation in randomized experiments using auxiliary observational data

Balanced versus Randomized Field Experiments in Economics: Why W. S. Gosset aka "Student" Matters

Design and analysis of experiments in networks: Reducing bias from interference

Why Experimenters Might Not Always Want to Randomize, and What They Could Do Instead

Equality, equity and incentives: An experiment

The Role of Randomized Field Trials in Social Science Research

Analysis of Randomized Experiments with Network Interference and Noncompliance

Nearly Random Designs with Greatly Improved Balance

Improving Covariate Balance in 2K Factorial Designs Via Rerandomization with an Application to a New York City Department of Education High School Study

Understanding an impact of patient enrollment pattern on predictability of central (unstratified) randomization in a multi‐center clinical trial