Abstract:Suppose an online platform wants to compare a treatment and control policy, e.g., two different matching algorithms in a ridesharing system, or two different inventory management algorithms in an online retail site. Standard randomized controlled trials are typically not feasible, since the goal is to estimate policy performance on the entire system. Instead, the typical current practice involves dynamically alternating between the two policies for fixed lengths of time, and comparing the average performance of each over the intervals in which they were run as an estimate of the treatment effect. However, this approach suffers from *temporal interference*: one algorithm alters the state of the system as seen by the second algorithm, biasing estimates of the treatment effect. Further, the simple non-adaptive nature of such designs implies they are not sample efficient. We develop a benchmark theoretical model in which to study optimal experimental design for this setting. We view testing the two policies as the problem of estimating the steady state difference in reward between two unknown Markov chains (i.e., policies). We assume estimation of the steady state reward for each chain proceeds via nonparametric maximum likelihood, and search for consistent (i.e., asymptotically unbiased) experimental designs that are efficient (i.e., asymptotically minimum variance). Characterizing such designs is equivalent to a Markov decision problem with a minimum variance objective; such problems generally do not admit tractable solutions. Remarkably, in our setting, using a novel application of classical martingale analysis of Markov chains via Poisson's equation, we characterize efficient designs via a succinct convex optimization problem. We use this characterization to propose a consistent, efficient online experimental design that adaptively samples the two Markov chains.

Policy evaluation for temporal and/or spatial dependent experiments

Policy Evaluation for Temporal and/or Spatial Dependent Experiments

Spatially Randomized Designs Can Enhance Policy Evaluation

Combining Experimental and Historical Data for Policy Evaluation

Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferences

A computational approach to identification of treatment effects for policy evaluation

Causal Modeling of Policy Interventions From Sequences of Treatments and Outcomes

Temporal-Spatial Entropy Balancing for Causal Continuous Treatment-Effect Estimation

A Semiparametric Instrumented Difference-in-Differences Approach to Policy Learning

A Bracketing Relationship for Long-Term Policy Evaluation with Combined Experimental and Observational Data

Estimating Direct and Indirect Causal Effects of Spatiotemporal Interventions in Presence of Spatial Interference

Learning Continuous Treatment Policy and Bipartite Embeddings for Matching with Heterogeneous Causal Effects

A Spatial Perspective on the Econometrics of Program Evaluation

Statistical Inference for Temporal Difference Learning with Linear Function Approximation

On Policy Evaluation with Aggregate Time-Series Shocks

A Causal Framework for Evaluating Heterogeneous Policy Mechanisms Using Difference-in-Differences

Fixed Effects Spatial Panel Data Models with Time-Varying Spatial Dependence

Difference-in-Differences for Policy Evaluation

Adaptive Experimental Design with Temporal Interference: A Maximum Likelihood Approach

Inference in Spatial Experiments with Interference using the SpatialEffect Package

Estimating heterogeneous treatment effects in nonstationary time series with state-space models