Abstract:Across domains such as medicine, employment, and criminal justice, predictive models often target labels that imperfectly reflect the outcomes of interest to experts and policymakers. For example, clinical risk assessments deployed to inform physician decision-making often predict measures of healthcare utilization (e.g., costs, hospitalization) as a proxy for patient medical need. These proxies can be subject to outcome measurement error when they systematically differ from the target outcome they are intended to measure. However, prior modeling efforts to characterize and mitigate outcome measurement error overlook the fact that the decision being informed by a model often serves as a risk-mitigating intervention that impacts the target outcome of interest and its recorded proxy. Thus, in these settings, addressing measurement error requires counterfactual modeling of treatment effects on outcomes. In this work, we study intersectional threats to model reliability introduced by outcome measurement error, treatment effects, and selection bias from historical decision-making policies. We develop an unbiased risk minimization method which, given knowledge of proxy measurement error properties, corrects for the combined effects of these challenges. We also develop a method for estimating treatment-dependent measurement error parameters when these are unknown in advance. We demonstrate the utility of our approach theoretically and via experiments on real-world data from randomized controlled trials conducted in healthcare and employment domains. As importantly, we demonstrate that models correcting for outcome measurement error or treatment effects alone suffer from considerable reliability limitations. Our work underscores the importance of considering intersectional threats to model validity during the design and evaluation of predictive models for decision support.

Learning Counterfactual Outcomes of MOOC Multiple Learning Behaviors

Counterfactual Adversarial Learning with Representation Interpolation

Modeling and Predicting Learning Behavior in MOOCs

Counterfactual Prediction for Outcome-oriented Treatments

Transfer Learning using Representation Learning in Massive Open Online Courses

An evaluation model based on procedural behaviors for predicting MOOC learning performance: students' online learning behavior analytics and algorithms construction

Learning Decomposed Representation for Counterfactual Inference

Information Theoretic Counterfactual Learning from Missing-Not-At-Random Feedback

Counterfactual Prediction Under Outcome Measurement Error

Multi-Objective Counterfactual Explanations

Optimization-based Causal Estimation from Heterogenous Environments

Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis

Counterfactual Generative Modeling with Variational Causal Inference

Predicting Cognitive Presence in At-Scale Online Learning: MOOC and For-Credit Online Course Environments

Counterfactual Fairness by Combining Factual and Counterfactual Predictions

Towards triggering higher-order thinking behaviors in MOOCs

An effect analysis of the balancing techniques on the counterfactual explanations of student success prediction models

Causal Inference Meets Machine Learning

The Promises of Parallel Outcomes

Stopout Research Based on Behavior in Massive Open Online Courses

Cycle-Balanced Representation Learning For Counterfactual Inference