Abstract:Suppose an online platform wants to compare a treatment and control policy, e.g., two different matching algorithms in a ridesharing system, or two different inventory management algorithms in an online retail site. Standard randomized controlled trials are typically not feasible, since the goal is to estimate policy performance on the entire system. Instead, the typical current practice involves dynamically alternating between the two policies for fixed lengths of time, and comparing the average performance of each over the intervals in which they were run as an estimate of the treatment effect. However, this approach suffers from *temporal interference*: one algorithm alters the state of the system as seen by the second algorithm, biasing estimates of the treatment effect. Further, the simple non-adaptive nature of such designs implies they are not sample efficient. We develop a benchmark theoretical model in which to study optimal experimental design for this setting. We view testing the two policies as the problem of estimating the steady state difference in reward between two unknown Markov chains (i.e., policies). We assume estimation of the steady state reward for each chain proceeds via nonparametric maximum likelihood, and search for consistent (i.e., asymptotically unbiased) experimental designs that are efficient (i.e., asymptotically minimum variance). Characterizing such designs is equivalent to a Markov decision problem with a minimum variance objective; such problems generally do not admit tractable solutions. Remarkably, in our setting, using a novel application of classical martingale analysis of Markov chains via Poisson's equation, we characterize efficient designs via a succinct convex optimization problem. We use this characterization to propose a consistent, efficient online experimental design that adaptively samples the two Markov chains.

Approximate cost-efficient sequential designs for binary response models with application to switching measurements

Experimental Designs for Binary Data in Switching Measurements on Superconducting Josephson Junctions

Dynamic programming principle in cost-efficient sequential design: application to switching measurements

D-optimal Approximate Design for Binary Regression and Quantal Response in Toxicology Studies

On the Relative Efficiency of Model-Assisted Designs: a Conditional Approach.

Efficient construction of Bayes optimal designs for stochastic process models

An adaptive approach to Bayesian Optimization with switching costs

Sequential minimum optimization algorithm with small sample size estimators

Sequential Computer Experimental Design for Estimating an Extreme Probability or Quantile

Sequential optimal experimental design for vapor-liquid equilibrium modeling

Sequential Bayesian optimal experimental design for structural reliability analysis

A reinforced learning approach to optimal design under model uncertainty

Adaptive Experimental Design with Temporal Interference: A Maximum Likelihood Approach

An Experimental Study of Human Decisions in Sequential Information Acquisition in Design: Impact of Cost and Task Complexity

LOCALLY D-OPTIMAL DESIGNS FOR HIERARCHICAL RESPONSE EXPERIMENTS

A Sequential Monte Carlo Algorithm to Incorporate Model Uncertainty in Bayesian Sequential Design

Evaluation of combinatorial optimisation algorithms for c-optimal experimental designs with correlated observations

Sequential design of single-cell experiments to identify discrete stochastic models for gene expression

Josephson Sampler Response Using a Binary Search Algorithm

Stochastic Model of Qudit Measurement for Superconducting Quantum Information Processing

Switching Time Optimization for Binary Quantum Optimal Control