Abstract:Across domains such as medicine, employment, and criminal justice, predictive models often target labels that imperfectly reflect the outcomes of interest to experts and policymakers. For example, clinical risk assessments deployed to inform physician decision-making often predict measures of healthcare utilization (e.g., costs, hospitalization) as a proxy for patient medical need. These proxies can be subject to outcome measurement error when they systematically differ from the target outcome they are intended to measure. However, prior modeling efforts to characterize and mitigate outcome measurement error overlook the fact that the decision being informed by a model often serves as a risk-mitigating intervention that impacts the target outcome of interest and its recorded proxy. Thus, in these settings, addressing measurement error requires counterfactual modeling of treatment effects on outcomes. In this work, we study intersectional threats to model reliability introduced by outcome measurement error, treatment effects, and selection bias from historical decision-making policies. We develop an unbiased risk minimization method which, given knowledge of proxy measurement error properties, corrects for the combined effects of these challenges. We also develop a method for estimating treatment-dependent measurement error parameters when these are unknown in advance. We demonstrate the utility of our approach theoretically and via experiments on real-world data from randomized controlled trials conducted in healthcare and employment domains. As importantly, we demonstrate that models correcting for outcome measurement error or treatment effects alone suffer from considerable reliability limitations. Our work underscores the importance of considering intersectional threats to model validity during the design and evaluation of predictive models for decision support.

Prediction in Measurement Error Models

Long-Term Prediction Intervals of Time Series

Robust Bayesian Inference for Berkson and Classical Measurement Error Models

Valid prediction intervals for regression problems

A general algorithm for error-in-variables regression modelling using Monte Carlo expectation maximization

Counterfactual Prediction Under Outcome Measurement Error

Predicting Conversion Time from Mild Cognitive Impairment to Dementia with Interval-Censored Models

Prediction interval: A powerful statistical tool for monitoring patients and analytical systems

Tight Prediction Intervals Using Expanded Interval Minimization

Inference in High-Dimensional Linear Measurement Error Models

Estimating the Confidence Interval for Prediction Errors of Support Vector Machine Classifiers

Prediction Accuracy Measures for Time-to-event Models with Left-Truncated and Right-Censored Data

Transformed Fay-Herriot Model with Measurement Error in Covariates

Explain Variance of Prediction in Variational Time Series Models for Clinical Deterioration Prediction

Prediction of future hospital admissions - what is the tradeoff between specificity and accuracy?

Valid predictions of random quantities in linear mixed models

Tolerance and Prediction Intervals for Non-normal Models

Difference-based estimation for error variances in repeated measurement regression models

Measurement Error as a Missing Data Problem

Variation in prediction accuracy due to randomness in data division and fair evaluation using interval estimation

A general, simple, robust method to account for measurement error when analyzing data with an internal validation subsample