Abstract:Randomized controlled trials (RCTs), or experiments, are the gold standard for intervention evaluation. However, the main appeal of RCTs, the clean identification of causal effects, can be compromised by interference, when one subject's treatment assignment can influence another subject's behavior or outcomes. In this paper, we formalise and study a type of interference stemming from the operational implementation of a subclass of interventions we term Service Interventions (SIs): interventions that include an on-demand service component provided by a costly and limited resource (e.g., healthcare providers or teachers). We show that in such a system, the capacity constraints induce dependencies across experiment subjects, where an individual may need to wait before receiving the intervention. By modeling these dependencies using a queueing system, we show how increasing the number of subjects without increasing the capacity of the system can lead to a nonlinear decrease in the treatment effect size. This has implications for conventional power analysis and recruitment strategies: increasing the sample size of an RCT without appropriately expanding capacity can decrease the study's power. To address this issue, we propose a method to jointly select the system capacity and number of users using the square root staffing rule from queueing theory. We show how incorporating knowledge of the queueing structure can help an experimenter reduce the amount of capacity and number of subjects required while still maintaining high power. In addition, our analysis of congestion-driven interference provides one concrete mechanism to explain why similar protocols can result in different RCT outcomes and why promising interventions at the RCT stage may not perform well at scale.

Randomized Controlled Trials of Service Interventions: The Impact of Capacity Constraints

Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects

Population Intervention Causal Effects Based on Stochastic Interventions

Experimenting under Stochastic Congestion

Randomization to Randomization Probability: Estimating Treatment Effects under Actual Conditions of Use.

Causal inference with limited resources: proportionally-representative interventions

The Micro-Randomized Trial for Developing Digital Interventions: Experimental Design Considerations

Identification of Single-Treatment Effects in Factorial Experiments

Beyond A/B Testing: Sequential Randomization for Developing Interventions in Scaled Digital Learning Environments

Analysis of Randomized Experiments with Network Interference and Noncompliance

SECRETS: Subject-Efficient Clinical Randomized Controlled Trials using Synthetic Intervention

Evaluating the Effectiveness of Index-Based Treatment Allocation

Assessing Health Care Interventions via an Interrupted Time Series Model: Study Power and Design Considerations

Improved Policy Evaluation for Randomized Trials of Algorithmic Resource Allocation

Staggered interventions with no control groups

Incorporating Contact Network Structure in Cluster Randomized Trials

A Reanalysis of Cluster Randomized Trials Showed Interrupted Time-Series Studies Were Valuable in Health System Evaluation.

Individualized treatment rules under stochastic treatment cost constraints

Barriers and enablers to conducting cluster randomized control trials in hospitals: A theory-informed scoping review

Stochastic interventions, sensitivity analysis, and optimal transport

Network Interference in Micro-Randomized Trials