Abstract:Across domains such as medicine, employment, and criminal justice, predictive models often target labels that imperfectly reflect the outcomes of interest to experts and policymakers. For example, clinical risk assessments deployed to inform physician decision-making often predict measures of healthcare utilization (e.g., costs, hospitalization) as a proxy for patient medical need. These proxies can be subject to outcome measurement error when they systematically differ from the target outcome they are intended to measure. However, prior modeling efforts to characterize and mitigate outcome measurement error overlook the fact that the decision being informed by a model often serves as a risk-mitigating intervention that impacts the target outcome of interest and its recorded proxy. Thus, in these settings, addressing measurement error requires counterfactual modeling of treatment effects on outcomes. In this work, we study intersectional threats to model reliability introduced by outcome measurement error, treatment effects, and selection bias from historical decision-making policies. We develop an unbiased risk minimization method which, given knowledge of proxy measurement error properties, corrects for the combined effects of these challenges. We also develop a method for estimating treatment-dependent measurement error parameters when these are unknown in advance. We demonstrate the utility of our approach theoretically and via experiments on real-world data from randomized controlled trials conducted in healthcare and employment domains. As importantly, we demonstrate that models correcting for outcome measurement error or treatment effects alone suffer from considerable reliability limitations. Our work underscores the importance of considering intersectional threats to model validity during the design and evaluation of predictive models for decision support.

Counterfactual Imputation: Comments on "Imputation of Counterfactual Outcomes when the Errors are Predictable" by Silvia Gonçalves and Serena Ng

Discussion of "Imputation of Counterfactual Outcomes when the Errors are Predictable" by Sílvia Gonçalves and Serena Ng

On "Imputation of Counterfactual Outcomes when the Errors are Predictable": Discussions on Misspecification and Suggestions of Sensitivity Analyses

Counterfactual Analysis and Inference With Nonstationary Data

Imputation of Counterfactual Outcomes when the Errors are Predictable

Imputation of Counterfactual Outcomes when the Errors are Predictable: Rejoinder

Conflating marginal and conditional treatment effects: Comments on 'Assessing the performance of population adjustment methods for anchored indirect comparisons: A simulation study'

A Practical Guide to Counterfactual Estimators for Causal Inference with Time-Series Cross-Sectional Data

Counterfactual Prediction Under Outcome Measurement Error

On Counterfactual Interventions in Vector Autoregressive Models

MIMICKING COUNTERFACTUAL OUTCOMES TO ESTIMATE CAUSAL EFFECTS

Counterfactual inference for sequential experiments

Distributional counterfactual analysis in high-dimensional setup

Estimating individual treatment effect: generalization bounds and algorithms

To Impute or not to Impute? Missing Data in Treatment Effect Estimation

Incentive-Aware Synthetic Control: Accurate Counterfactual Estimation via Incentivized Exploration

Counterfactuals for the Future

Counterfactual: An R Package for Counterfactual Analysis

An Optimal Transport Approach to Estimating Causal Effects via Nonlinear Difference-in-Differences

Mission Imputable: Correcting for Berkson Error When Imputing a Censored Covariate