Abstract:A/B tests, also known as randomized controlled experiments (RCTs), are the gold standard for evaluating the impact of new policies, products, or decisions. However, these tests can be costly in terms of time and resources, potentially exposing users, customers, or other test subjects (units) to inferior options. This paper explores practical considerations in applying methodologies inspired by "synthetic control" as an alternative to traditional A/B testing in settings with very large numbers of units, involving up to hundreds of millions of units, which is common in modern applications such as e-commerce and ride-sharing platforms. This method is particularly valuable in settings where the treatment affects only a subset of units, leaving many units unaffected. In these scenarios, synthetic control methods leverage data from unaffected units to estimate counterfactual outcomes for treated units. After the treatment is implemented, these estimates can be compared to actual outcomes to measure the treatment effect. A key challenge in creating accurate counterfactual outcomes is interpolation bias, a well-documented phenomenon that occurs when control units differ significantly from treated units. To address this, we propose a two-phase approach: first using nearest neighbor matching based on unit covariates to select similar control units, then applying supervised learning methods suitable for high-dimensional data to estimate counterfactual outcomes. Testing using six large-scale experiments demonstrates that this approach successfully improves estimate accuracy. However, our analysis reveals that machine learning bias -- which arises from methods that trade off bias for variance reduction -- can impact results and affect conclusions about treatment effects. We document this bias in large-scale experimental settings and propose effective de-biasing techniques to address this challenge.

Policy Learning with Rare Outcomes

Machine learning in policy evaluation: new tools for causal inference

Policy Learning for Many Outcomes of Interest: Combining Optimal Policy Trees with Multi-objective Bayesian Optimisation

CATE meets ML -- The Conditional Average Treatment Effect and Machine Learning

Matching-Based Policy Learning

Learning treatment effects while treating those in need

Reduced-Rank Multi-objective Policy Learning and Optimization

Heterogeneous treatment effect estimation with high-dimensional data in public policy evaluation -- an application to the conditioning of cash transfers in Morocco using causal machine learning

More Efficient Policy Learning via Optimal Retargeting

Policy Learning with New Treatments

Estimating individual treatment effect: generalization bounds and algorithms

Policy Learning with Asymmetric Counterfactual Utilities

Pareto-Optimal Estimation and Policy Learning on Short-term and Long-term Treatment Effects

Policy Learning with Distributional Welfare

Evaluating Conditional Cash Transfer Policies with Machine Learning Methods

Post Launch Evaluation of Policies in a High-Dimensional Setting

Policy Learning with Competing Agents

Trustworthy Policy Learning under the Counterfactual No-Harm Criterion.

Optimal and Fair Encouragement Policy Evaluation and Learning

Robust Causal Learning for the Estimation of Average Treatment Effects

Learning Continuous Treatment Policy and Bipartite Embeddings for Matching with Heterogeneous Causal Effects