Abstract:Objective: Existing approaches to fairness evaluation often overlook systematic differences in the social determinants of health, like demographics and socioeconomics, among comparison groups, potentially leading to inaccurate or even contradictory conclusions. This study aims to evaluate racial disparities in predicting mortality among patients with chronic diseases using a fairness detection method that considers systematic differences. Methods: We created five datasets from Mass General Brigham's electronic health records (EHR), each focusing on a different chronic condition: congestive heart failure (CHF), chronic kidney disease (CKD), chronic obstructive pulmonary disease (COPD), chronic liver disease (CLD), and dementia. For each dataset, we developed separate machine learning models to predict 1-year mortality and examined racial disparities by comparing prediction performances between Black and White individuals. We compared racial fairness evaluation between the overall Black and White individuals versus their counterparts who were Black and matched White individuals identified by propensity score matching, where the systematic differences were mitigated. Results: We identified significant differences between Black and White individuals in age, gender, marital status, education level, smoking status, health insurance type, body mass index, and Charlson comorbidity index (p-value < 0.001). When examining matched Black and White subpopulations identified through propensity score matching, significant differences between particular covariates existed. We observed weaker significance levels in the CHF cohort for insurance type (p = 0.043), in the CKD cohort for insurance type (p = 0.005) and education level (p = 0.016), and in the dementia cohort for body mass index (p = 0.041); with no significant differences for other covariates. When examining mortality prediction models across the five study cohorts, we conducted a comparison of fairness evaluations before and after mitigating systematic differences. We revealed significant differences in the CHF cohort with p-values of 0.021 and 0.001 in terms of F1 measure and Sensitivity for the AdaBoost model, and p-values of 0.014 and 0.003 in terms of F1 measure and Sensitivity for the MLP model, respectively. Discussion and conclusion: This study contributes to research on fairness assessment by focusing on the examination of systematic disparities and underscores the potential for revealing racial bias in machine learning models used in clinical settings.

Methodological concerns about 'concordance-statistic for benefit' as a measure of discrimination in treatment benefit prediction

Unmasking Bias: A Framework for Evaluating Treatment Benefit Predictors Using Observational Studies

Decision Curve Analysis: a Technical Note

Assessing an Alternative for `Negative Variance Components': A Gentle Introduction to Bayesian Covariance Structure Modelling for Negative Associations Among Patients with Personalized Treatments

Quantifying Individual Risk for Binary Outcome: Bounds and Inference

Comparing methods for estimation of heterogeneous treatment effects using observational data from health care databases

Why Is My Classifier Discriminatory?

Weighted Brier Score -- an Overall Summary Measure for Risk Prediction Models with Clinical Utility Consideration

Predicting Counterfactuals from Large Historical Data and Small Randomized Trials

Can the potential benefit of individualizing treatment be assessed using trial summary statistics alone?

Counterfactual Prediction Under Outcome Measurement Error

Intersectional consequences for marginal fairness in prediction models of emergency admissions

Treatment Effect Estimation Via Differentiated Confounder Balancing and Regression

A Flexible Approach for Predictive Biomarker Discovery

A Threshold-free Prospective Prediction Accuracy Measure for Censored Time to Event Data

Comparison of methods for early-readmission prediction in a high-dimensional heterogeneous covariates and time-to-event outcome framework

Detecting critical treatment effect bias in small subgroups

GRADE Concept Paper 8: Judging the certainty of discrimination performance estimates of prognostic models in a body of validation studies

Conformal Inference of Counterfactuals and Individual Treatment Effects

Assessing fairness in machine learning models: A study of racial bias using matched counterparts in mortality prediction for patients with chronic diseases

Target specification bias, counterfactual prediction, and algorithmic fairness in healthcare