Interference Among First-Price Pacing Equilibria: A Bias and Variance Analysis

Luofeng Liao,Christian Kroer,Sergei Leonenkov,Okke Schrijvers,Liang Shi,Nicolas Stier-Moses,Congshan Zhang
2024-02-12
Abstract:Online A/B testing is widely used in the internet industry to inform decisions on new feature roll-outs. For online marketplaces (such as advertising markets), standard approaches to A/B testing may lead to biased results when buyers operate under a budget constraint, as budget consumption in one arm of the experiment impacts performance of the other arm. To counteract this interference, one can use a budget-split design where the budget constraint operates on a per-arm basis and each arm receives an equal fraction of the budget, leading to ``budget-controlled A/B testing.'' Despite clear advantages of budget-controlled A/B testing, performance degrades when budget are split too small, limiting the overall throughput of such systems. In this paper, we propose a parallel budget-controlled A/B testing design where we use market segmentation to identify submarkets in the larger market, and we run parallel experiments on each submarket.
Statistics Theory,Computer Science and Game Theory,Econometrics
What problem does this paper attempt to address?