Abstract:One of the central difficulties of addressing the COVID-19 pandemic has been accurately measuring and predicting the spread of infections. In particular, official COVID-19 case counts in the United States are under counts of actual caseloads due to the absence of universal testing policies. Researchers have proposed a variety of methods for recovering true caseloads, often through the estimation of statistical models on more reliable measures, such as death and hospitalization counts, positivity rates, and demographics. However, given the disproportionate impact of COVID-19 on marginalized racial, ethnic, and socioeconomic groups, it is important to consider potential unintended effects of case correction methods on these groups. Thus, we investigate two of these correction methods for their impact on a downstream COVID-19 case prediction task. For that purpose, we tailor an auditing approach and evaluation protocol to analyze the fairness of the COVID-19 prediction task by measuring the difference in model performance between majority-White counties and majority-minority counties. We find that one of the correction methods improves fairness, decreasing differences in performance between majority-White and majority-minority counties, while the other method increases differences, introducing bias. While these results are mixed, it is evident that correction methods have the potential to exacerbate existing biases in COVID-19 case data and in downstream prediction tasks. Researchers planning to develop or use case correction methods must be careful to consider negative effects on marginalized groups.

A Cautionary Tail: A Framework and Case Study for Testing Predictive Model Validity

A Case Study on a Sustainable Framework for Ethically Aware Predictive Modeling

Clinical Prediction Models: Model Validation

Consistent Validation for Predictive Methods in Spatial Settings

Predictive models of safety based on audit findings: Part 1: Model development and reliability.

On (in)validating environmental models. 1. Principles for formulating a Turing‐like Test for determining when a model is fit‐for purpose

Testing Causality in Scientific Modelling Software

Evaluating and Correcting Performative Effects of Decision Support Systems via Causal Domain Shift

Child welfare predictive risk models and legal decision making

No Free Delivery Service: Epistemic limits of passive data collection in complex social systems

Hindsight Analysis of the Chicago Food Inspection Forecasting Model

Pursuing a Prospective Perspective

A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes

Predictive risk modeling for child maltreatment detection and enhanced decision-making: Evidence from Danish administrative data

The Misuse of AUC: What High Impact Risk Assessment Gets Wrong

Evaluation of clinical prediction models (part 2): how to undertake an external validation study

Statistical Development and Validation of Clinical Prediction Models

Assessing the Impact of Case Correction Methods on the Fairness of COVID-19 Predictive Models

Assessing Spatial Predictive Models in the Environmental Sciences: Accuracy Measures, Data Variation and Variance Explained

Assessing the performance of spatial cross-validation approaches for models of spatially structured data

Selection tests work better than we think they do, and have for years