Abstract:Online marketplace designers frequently run A/B tests to measure the impact of proposed product changes. However, given that marketplaces are inherently connected, total average treatment effect estimates obtained through Bernoulli randomized experiments are often biased due to violations of the stable unit treatment value assumption. This can be particularly problematic for experiments that impact sellers' strategic choices, affect buyers' preferences over items in their consideration set, or change buyers' consideration sets altogether. In this work, we measure and reduce bias due to interference in online marketplace experiments by using observational data to create clusters of similar listings, and then using those clusters to conduct cluster-randomized field experiments. We provide a lower bound on the magnitude of bias due to interference by conducting a meta-experiment that randomizes over two experiment designs: one Bernoulli randomized, one cluster randomized. In both meta-experiment arms, treatment sellers are subject to a different platform fee policy than control sellers, resulting in different prices for buyers. By conducting a joint analysis of the two meta-experiment arms, we find a large and statistically significant difference between the total average treatment effect estimates obtained with the two designs, and estimate that 32.60% of the Bernoulli-randomized treatment effect estimate is due to interference bias. We also find weak evidence that the magnitude and/or direction of interference bias depends on extent to which a marketplace is supply- or demand-constrained, and analyze a second meta-experiment to highlight the difficulty of detecting interference bias when treatment interventions require intention-to-treat analysis.

Testing the effectiveness of lottery incentives in online experiments

Why High Incentives Cause Repugnance: A Framed Field Experiment

An experimental test of whether financial incentives constitute undue inducement in decision-making

USING LOTTERIES TO FINANCE PUBLIC GOODS: THEORY AND EXPERIMENTAL EVIDENCE

Reducing Interference Bias in Online Marketplace Pricing Experiments

Reduction of compound lotteries with objective probabilities: Theory and evidence

The effect of incentives on intertemporal choice: Choice, confidence, and eye movements

Hyper-altruistic behavior vanishes with high stakes

Are non-fungible payments attractive when they reduce risk exposure? Evidence from Colombia

Guaranteed Incentives and Prize Drawings: Effects on Participation, Data Quality, and Costs in a Web Survey of College Students on Sensitive Topics

The Online Laboratory: Conducting Experiments in a Real Labor Market

Equality, equity and incentives: An experiment

Experimental evidence on the effect of incentives and domain in risk aversion and discounting tasks

Online Fund-Raising Mechanisms: A Field Experiment

Eat & Tell: A Randomized Trial of Random-Loss Incentive to Increase Dietary Self-Tracking Compliance

The Long-Term Impact of Matching and Rebate Subsidies When Public Goods are Impure: Field Experimental Evidence from the Carbon Offsetting Market

Sports betting incentives encourage gamblers to select the long odds: An experimental investigation using monetary rewards

Cash or Non-Cash? Unveiling Ideators' Incentive Preferences in Crowdsourcing Contests

Incentives for Blood Donation: A Discrete Choice Experiment to Analyze Extrinsic Motivation

Incentives modulate arousal and attention in risky choice

Gambling in risk-taking contests: Experimental evidence