Abstract:Analyses of heterogeneous treatment effects (HTE) are common in applied causal inference research. However, when outcomes are latent variables assessed via psychometric instruments such as educational tests, standard methods ignore the potential HTE that may exist among the individual items of the outcome measure. Failing to account for ``item-level'' HTE (IL-HTE) can lead to both estimated standard errors that are too small and identification challenges in the estimation of treatment-by-covariate interaction effects. We demonstrate how Item Response Theory (IRT) models that estimate a treatment effect for each assessment item can both address these challenges and provide new insights into HTE generally. This study articulates the theoretical rationale for the IL-HTE model and demonstrates its practical value using 73 data sets from 46 randomized controlled trials containing 5.8 million item responses in economics, education, and health research. Our results show that the IL-HTE model reveals item-level variation masked by single-number scores, provides more meaningful standard errors in many settings, allows for estimates of the generalizability of causal effects to untested items, resolves identification problems in the estimation of interaction effects, and provides estimates of standardized treatment effect sizes corrected for attenuation due to measurement error.

What problem does this paper attempt to address?

### Problems the paper attempts to solve This paper aims to solve the problem that in causal inference research, when the outcome variable is a latent variable evaluated by a psychometric tool (such as educational tests, psychological surveys, or patient - self - reported disease symptoms), the standard methods ignore the possible Item - Level Heterogeneous Treatment Effects (IL - HTE) among individual items. Specifically, the paper points out: 1. **The impact of ignoring IL - HTE**: Traditional treatment - effect analysis methods usually ignore the differences in the sensitivity of individual items to treatment effects, which may lead to an overly small estimated standard error and identification problems when estimating the interaction effects between treatment and covariates. 2. **Theoretical and empirical value**: The paper solves the above problems by introducing the Item Response Theory (IRT) model to estimate the treatment effect of each item. These models can not only provide more accurate standard errors but also reveal the item - level variation masked by a single numerical score. 3. **Wide applicability**: The paper uses 73 datasets from 46 randomized controlled trials (RCTs), including 5.8 million item responses, covering the fields of economics, education, and health research. Through these data, the paper demonstrates the application value of the IL - HTE model in five aspects: - **Explaining the variation of item - specific or sub - scale effects**: The IL - HTE model provides an interpretable measure of the variation of item - specific effects. - **Considering the uncertainty of item sampling**: The IL - HTE model can flexibly consider the uncertainty brought by randomly sampling items from a broader item pool. - **Generalization estimation of untested items**: The IL - HTE model can estimate the treatment effects of untested items through prediction intervals. - **Distinguishing the heterogeneity of item characteristics and individual characteristics**: The IL - HTE model can distinguish the treatment - effect heterogeneity that depends on item characteristics from that which depends on individual characteristics. - **Correcting the attenuation of effect size caused by measurement error**: The IL - HTE model provides estimates of standardized effect sizes that have been corrected for the attenuation due to measurement error. ### Formula presentation - **Potential outcome framework**: \[ \theta_j=\beta_0 + \beta_1T_j+\epsilon_j \] where \(\theta_j\) is the potential outcome of the \(j\) - th person, \(T_j\) is the treatment - status indicator variable, \(\beta_0\) is the mean of the control group, \(\beta_1\) is the treatment effect, and \(\epsilon_j\) is the error term. - **Item response theory model**: \[ \logit(P(Y_{ij} = 1))=\eta_{ij}=\theta_j + b_i+\zeta_iT_j \] \[ \theta_j=\beta_0 + \beta_1T_j+\epsilon_j \] \[ \begin{bmatrix} b_i\\ \zeta_i \end{bmatrix}\sim N\left(\begin{bmatrix} 0\\ 0 \end{bmatrix},\begin{bmatrix} \sigma_b^2&\rho\sigma_b\sigma_\zeta\\ \rho\sigma_b\sigma_\zeta&\sigma_\zeta^2 \end{bmatrix}\right) \] \[ \epsilon_j\sim N(0,\sigma_\theta^2) \] where \(\zeta_i\) represents the residual treatment effect of the \(i\) - th item, and \(\sigma_\zeta\) is the item - specific treatment effect.

Estimating Heterogeneous Treatment Effects with Item-Level Outcome Data: Insights from Item Response Theory

Modeling Item-Level Heterogeneous Treatment Effects With the Explanatory Item Response Model: Leveraging Large-Scale Online Assessments to Pinpoint the Impact of Educational Interventions

Identifying Heterogeneous Treatment Effects in Multiple Outcomes using Joint Confidence Intervals

Item response theory in high-stakes pharmacy assessments

Identification and Estimation of Heterogeneous Treatment Effects under Non-compliance or Non-ignorable assignment

Estimating individual treatment effect: generalization bounds and algorithms

Testing Whether Reported Treatment Effects are Unduly Dependent on the Specific Outcome Measure Used

A Systematic Paradigm for Detecting, Surfacing, and Characterizing Heterogeneous Treatment Effects (HTE)

Lifting the veil off treatment effect heterogeneity

Item Response Theory -- A Statistical Framework for Educational and Psychological Measurement

Uncovering individualised treatment effects for educational trials

Heterogeneous Treatment Effects in the Presence of Self-Selection: A Propensity Score Perspective

Budgeted Heterogeneous Treatment Effect Estimation

Estimating Heterogeneous Treatment Effects with Observational Data

Evaluation of covariate effects in item response theory models

Estimation of treatment effects under endogeneous heteroskedasticity

Benefits of the Curious Behavior of Bayesian Hierarchical Item Response Theory Models—An in-Depth Investigation and Bias Correction

A study of alternative approaches to non-normal latent trait distributions in item response theory models used for health outcome measurement

Detecting Heterogeneous Treatment Effect with Instrumental Variables

Stable Estimation of Heterogeneous Treatment Effects.

Optimal Statistical Inference for Individualized Treatment Effects in High-dimensional Models