Abstract:Across domains such as medicine, employment, and criminal justice, predictive models often target labels that imperfectly reflect the outcomes of interest to experts and policymakers. For example, clinical risk assessments deployed to inform physician decision-making often predict measures of healthcare utilization (e.g., costs, hospitalization) as a proxy for patient medical need. These proxies can be subject to outcome measurement error when they systematically differ from the target outcome they are intended to measure. However, prior modeling efforts to characterize and mitigate outcome measurement error overlook the fact that the decision being informed by a model often serves as a risk-mitigating intervention that impacts the target outcome of interest and its recorded proxy. Thus, in these settings, addressing measurement error requires counterfactual modeling of treatment effects on outcomes. In this work, we study intersectional threats to model reliability introduced by outcome measurement error, treatment effects, and selection bias from historical decision-making policies. We develop an unbiased risk minimization method which, given knowledge of proxy measurement error properties, corrects for the combined effects of these challenges. We also develop a method for estimating treatment-dependent measurement error parameters when these are unknown in advance. We demonstrate the utility of our approach theoretically and via experiments on real-world data from randomized controlled trials conducted in healthcare and employment domains. As importantly, we demonstrate that models correcting for outcome measurement error or treatment effects alone suffer from considerable reliability limitations. Our work underscores the importance of considering intersectional threats to model validity during the design and evaluation of predictive models for decision support.

Clinical decision making under uncertainty: a bootstrapped counterfactual inference approach

Uncertainty-Aware Optimal Treatment Selection for Clinical Time Series

Policy Learning with Asymmetric Counterfactual Utilities

Conformal Counterfactual Inference under Hidden Confounding

Decision Curve Analysis: a Technical Note

Cost-Effective Incentive Allocation via Structured Counterfactual Inference

Counterfactual Prediction Under Outcome Measurement Error

Predictive Performance Comparison of Decision Policies Under Confounding

Model-Assisted Uniformly Honest Inference for Optimal Treatment Regimes in High Dimension

Bayesian Decision Curve Analysis With Bayesdca

Decision-Focused Uncertainty Quantification

An optimal learning method for developing personalized treatment regimes

Identifying Causal-Effect Inference Failure with Uncertainty-Aware Models

A Case Study of Stochastic Optimization in Health Policy: Problem Formulation and Preliminary Results

Counterfactually Guided Off-policy Transfer in Clinical Settings

Designing Decision Support Systems Using Counterfactual Prediction Sets

Algorithmic Decision Making in the Presence of Unmeasured Confounding

Learning to Bound Counterfactual Inference from Observational, Biased and Randomised Data

Stochastic Intervention for Causal Inference via Reinforcement Learning

Counterfactual Reasoning for Fair Clinical Risk Prediction

Leveraging Clinical Time-Series Data for Prediction: A Cautionary Tale