Abstract:Randomized experiments, or A/B tests are used to estimate the causal impact of a feature on the behavior of users by creating two parallel universes in which members are simultaneously assigned to treatment and control. However, in social network settings, members interact, such that the impact of a feature is not always contained within the treatment group. Researchers have developed a number of experimental designs to estimate network effects in social settings. Alternatively, naturally occurring exogenous variation, or 'natural experiments,' allow researchers to recover causal estimates of peer effects from observational data in the absence of experimental manipulation. Natural experiments trade off the engineering costs and some of the ethical concerns associated with network randomization with the search costs of finding situations with natural exogenous variation. To mitigate the search costs associated with discovering natural counterfactuals, we identify a common engineering requirement used to scale massive online systems, in which natural exogenous variation is likely to exist: notification queueing. We identify two natural experiments on the LinkedIn platform based on the order of notification queues to estimate the causal impact of a received message on the engagement of a recipient. We show that receiving a message from another member significantly increases a member's engagement, but that some popular observational specifications, such as fixed-effects estimators, overestimate this effect by as much as 2.7x. We then apply the estimated network effect coefficients to a large body of past experiments to quantify the extent to which it changes our interpretation of experimental results. The study points to the benefits of using messaging queues to discover naturally occurring counterfactuals for the estimation of causal effects without experimenter intervention.

Randomized experiments to detect and estimate social influence in networks

Design and analysis of experiments in networks: Reducing bias from interference

Estimating Total Treatment Effect in Randomized Experiments with Unknown Network Structure

Design-based inference for generalized network experiments with stochastic interventions

Causal Inference Under Network Interference: A Framework for Experiments on Social Networks

Estimating Network Effects Using Naturally Occurring Peer Notification Queue Counterfactuals

A Simple Model to Estimate Sharing Effects in Social Networks

Evaluating stochastic seeding strategies in networks

Causal message-passing for experiments with unknown and general network interference

Causal Inference under Network Interference Using a Mixture of Randomized Experiments

Randomized Experiments in Education, with Implications for Multilevel Causal Inference

Estimating Social Influence from Observational Data

Estimation and inference for causal spillover effects in egocentric-network randomized trials in the presence of network membership misclassification

Independent-Set Design of Experiments for Estimating Treatment and Spillover Effects under Network Interference

Cascade-based Randomization for Inferring Causal Effects under Diffusion Interference

Causal Effects with Hidden Treatment Diffusion on Observed or Partially Observed Networks

Exploiting Neighborhood Interference with Low Order Interactions under Unit Randomized Design

Tie Strength, Embeddedness, and Social Influence: A Large-Scale Networked Experiment

Design-based Estimation Theory for Complex Experiments

Controllability of social networks and the strategic use of random information

Causal Message Passing: A Method for Experiments with Unknown and General Network Interference