Abstract:Online controlled experiments play a crucial role in enabling data-driven decisions across a wide range of companies. Variance reduction is an effective technique to improve the sensitivity of experiments, achieving higher statistical power while using fewer samples and shorter experimental periods. However, typical variance reduction methods (e.g., regression-adjusted estimators) are built upon the intuitional assumption of Gaussian distributions and cannot properly characterize the real business metrics with heavy-tailed distributions. Furthermore, outliers diminish the correlation between pre-experiment covariates and outcome metrics, greatly limiting the effectiveness of variance reduction. In this paper, we develop a novel framework that integrates the Student's t-distribution with machine learning tools to fit heavy-tailed metrics and construct a robust average treatment effect estimator in online controlled experiments, which we call STATE. By adopting a variational EM method to optimize the loglikehood function, we can infer a robust solution that greatly eliminates the negative impact of outliers and achieves significant variance reduction. Moreover, we extend the STATE method from count metrics to ratio metrics by utilizing linear transformation that preserves unbiased estimation, whose variance reduction is more complex but less investigated in existing works. Finally, both simulations on synthetic data and long-term empirical results on Meituan experiment platform demonstrate the effectiveness of our method. Compared with the state-of-the-art estimators (CUPAC/MLRATE), STATE achieves over 50% variance reduction, indicating it can reach the same statistical power with only half of the observations, or half the experimental duration.

Global Rank Sum Test: An Efficient Rank-Based Nonparametric Test for Large Scale Online Experiment

Large-Scale Online Experimentation with Quantile Metrics

Multivariate Rank-Based Analysis of Multiple Endpoints in Clinical Trials: A Global Test Approach

Inference at Scale Significance Testing for Large Search and Recommendation Experiments

An Online Sequential Test for Qualitative Treatment Effects

Heterocedasticity-Adjusted Ranking and Thresholding for Large-Scale Multiple Testing

Rank-transformed subsampling: inference for multiple data splitting and exchangeable p-values

Equivalence Test in Multi-dimensional Space with Applications in A/B Testing

Trustworthy Online Marketplace Experimentation with Budget-split Design

Power Comparison between High Dimensional t-Test, Sign, and Signed Rank Tests

Online Experimentation with Surrogate Metrics: Guidelines and a Case Study

A New Procedure for Controlling False Discovery Rate in Large-Scale t-tests

A Rank Sum Test Method for Informative Gene Discovery.

Near-Optimal Experimental Design under the Budget Constraint in Online Platforms.

STATE: A Robust ATE Estimator of Heavy-Tailed Metrics for Variance Reduction in Online Controlled Experiments

Adaptive Testing for High-dimensional Data

Universal Rank Inference via Residual Subsampling with Application to Large Networks

A New Framework for Online Testing of Heterogeneous Treatment Effect

Testing two variances for superiority/non-inferiority and equivalence: Using the exhaustion algorithm for sample size allocation with cost

An Extreme-Value Approach for Testing the Equality of Large U-Statistic Based Correlation Matrices

Semiparametrically Point-Optimal Hybrid Rank Tests for Unit Roots