Testing weak nulls in matched observational studies

Colin B. Fogarty

DOI: https://doi.org/10.48550/arXiv.1908.07352

2020-02-16

Abstract:We develop sensitivity analyses for weak nulls in matched observational studies while allowing unit-level treatment effects to vary. The methods may be applied to studies using any optimal without-replacement matching algorithm. In contrast to randomized experiments and to paired observational studies, we show for general matched designs that over a large class of test statistics, any valid sensitivity analysis for the entirety of the weak null must be unnecessarily conservative if Fisher's sharp null of no treatment effect for any individual also holds. We present a sensitivity analysis valid for the weak null, and illustrate why it is generally conservative if the sharp null holds through new connections to inverse probability weighted estimators. An alternative procedure is presented that is asymptotically sharp if treatment effects are constant, and that is valid for the weak null under additional restrictions which may be deemed benign by practitioners. Simulations demonstrate that this alternative procedure results in a valid sensitivity analysis for the weak null hypothesis under a host of reasonable data-generating processes. The procedures allow practitioners to assess robustness of estimated sample average treatment effects to hidden bias while allowing for unspecified effect heterogeneity in matched observational studies.

Methodology

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to conduct an effective sensitivity analysis of the weak null hypothesis in matching observational studies. Specifically, the author focuses on how to conduct an effective sensitivity analysis of the weak null hypothesis (that is, the average treatment effect equals a certain value) when allowing for the existence of heterogeneity in individual - level treatment effects. Traditional sensitivity analysis methods usually assume that the treatment effect is constant, which may be too strict in practical applications because the treatment effects of different individuals may vary. Therefore, the goal of the paper is to develop a new sensitivity analysis method that can still effectively evaluate the impact of hidden bias on research results in the presence of treatment effect heterogeneity.

### Main contributions
1. **Theoretical framework**: The paper first shows that in general matching designs, any effective sensitivity analysis method based on the assumption of constant treatment effect cannot strictly control the expected value in the worst - case scenario when treatment effect heterogeneity exists. This means that traditional sensitivity analysis methods may be too conservative when treatment effect heterogeneity exists.
2. **New methods**: The paper proposes two new sensitivity analysis methods:
- The first method is always valid under the weak null hypothesis, but may be too conservative under the assumption of constant treatment effect.
- The second method can tightly control the expected value under the assumption of constant treatment effect and is still valid under the weak null hypothesis under some reasonable data - generation mechanisms.
3. **Practical applications**: Through simulation studies and real - data examples, the paper shows the effectiveness of these new methods in evaluating the impact of hidden bias on estimating the sample average treatment effect.

### Formula explanations
- **Treatment effect**: The individual - level treatment effect is represented as \(\tau_{ij} = r_{Tij} - r_{Cij}\), where \(r_{Tij}\) and \(r_{Cij}\) are the potential outcomes of the \(j\) - th individual in the \(i\) - th matching group under the treatment and control conditions respectively.
- **Sample average treatment effect**: \(\bar{\tau} = N^{-1} \sum_{i = 1}^B \sum_{j = 1}^{n_i} \tau_{ij}\), where \(N\) is the total sample size and \(n_i\) is the number of individuals in the \(i\) - th matching group.
- **Test statistic**: The paper uses multiple test statistics, such as \(\hat{\tau}_i=\sum_{j = 1}^{n_i} \left( Z_{ij} r_{Tij}-(1 - Z_{ij}) r_{Cij}/(n_i - 1) \right)\), where \(Z_{ij}\) is an indicator variable indicating whether the \(j\) - th individual in the \(i\) - th matching group has received the treatment.

### Conclusion
Through theoretical analysis and empirical research, the paper proposes a method for effectively conducting sensitivity analysis of the weak null hypothesis in the presence of treatment effect heterogeneity. These methods are not only of great theoretical significance but also show good performance in practical applications, providing researchers with a new tool for evaluating the impact of hidden bias.