Abstract:Recent years have experienced increasing utilization of complex machine learning models across multiple sources of data to inform more generalizable decision-making. However, distribution shifts across data sources and privacy concerns related to sharing individual-level data, coupled with a lack of uncertainty quantification from machine learning predictions, make it challenging to achieve valid inferences in multi-source environments. In this paper, we consider the problem of obtaining distribution-free prediction intervals for a target population, leveraging multiple potentially biased data sources. We derive the efficient influence functions for the quantiles of unobserved outcomes in the target and source populations, and show that one can incorporate machine learning prediction algorithms in the estimation of nuisance functions while still achieving parametric rates of convergence to nominal coverage probabilities. Moreover, when conditional outcome invariance is violated, we propose a data-adaptive strategy to upweight informative data sources for efficiency gain and downweight non-informative data sources for bias reduction. We highlight the robustness and efficiency of our proposals for a variety of conformal scores and data-generating mechanisms via extensive synthetic experiments. Hospital length of stay prediction intervals for pediatric patients undergoing a high-risk cardiac surgical procedure between 2016-2022 in the U.S. illustrate the utility of our methodology.

Distribution-Free Conditional Median Inference

Inference for Median and a Generalization of HulC

Distribution-Free Prediction Intervals Under Covariate Shift, With an Application to Causal Inference

Confidence Distribution and Distribution Estimation for Modern Statistical Inference

Multi-Source Conformal Inference Under Distribution Shift

Conformal Prediction using Conditional Histograms

Distribution-Free Inference for the Regression Function of Binary Classification

A Distribution Free Conditional Independence Test with Applications to Causal Discovery

Not all distributional shifts are equal: Fine-grained robust conformal inference

Distribution-free inference with hierarchical data

A nonparametric two-sample conditional distribution test

Distance and Kernel-Based Measures for Global and Local Two-Sample Conditional Distribution Testing

Distribution Free Prediction Bands

Conditional Diffusion Models are Minimax-Optimal and Manifold-Adaptive for Conditional Distribution Estimation

Conformal Inference for Invariant Risk Minimization

Distribution-free risk assessment of regression-based machine learning algorithms

Distribution-uniform anytime-valid sequential inference

Confidence distribution (CD) -- distribution estimator of a parameter

Confidence intervals for median absolute deviations

Asymptotics for conformal inference

Probabilistic Conformal Prediction with Approximate Conditional Validity