Abstract:Objective: Existing approaches to fairness evaluation often overlook systematic differences in the social determinants of health, like demographics and socioeconomics, among comparison groups, potentially leading to inaccurate or even contradictory conclusions. This study aims to evaluate racial disparities in predicting mortality among patients with chronic diseases using a fairness detection method that considers systematic differences. Methods: We created five datasets from Mass General Brigham's electronic health records (EHR), each focusing on a different chronic condition: congestive heart failure (CHF), chronic kidney disease (CKD), chronic obstructive pulmonary disease (COPD), chronic liver disease (CLD), and dementia. For each dataset, we developed separate machine learning models to predict 1-year mortality and examined racial disparities by comparing prediction performances between Black and White individuals. We compared racial fairness evaluation between the overall Black and White individuals versus their counterparts who were Black and matched White individuals identified by propensity score matching, where the systematic differences were mitigated. Results: We identified significant differences between Black and White individuals in age, gender, marital status, education level, smoking status, health insurance type, body mass index, and Charlson comorbidity index (p-value < 0.001). When examining matched Black and White subpopulations identified through propensity score matching, significant differences between particular covariates existed. We observed weaker significance levels in the CHF cohort for insurance type (p = 0.043), in the CKD cohort for insurance type (p = 0.005) and education level (p = 0.016), and in the dementia cohort for body mass index (p = 0.041); with no significant differences for other covariates. When examining mortality prediction models across the five study cohorts, we conducted a comparison of fairness evaluations before and after mitigating systematic differences. We revealed significant differences in the CHF cohort with p-values of 0.021 and 0.001 in terms of F1 measure and Sensitivity for the AdaBoost model, and p-values of 0.014 and 0.003 in terms of F1 measure and Sensitivity for the MLP model, respectively. Discussion and conclusion: This study contributes to research on fairness assessment by focusing on the examination of systematic disparities and underscores the potential for revealing racial bias in machine learning models used in clinical settings.

Retiring Adult: New Datasets for Fair Machine Learning

A Suite of Fairness Datasets for Tabular Classification

Lazy Data Practices Harm Fairness Research

AGR: Age Group fairness Reward for Bias Mitigation in LLMs

Optimisation Strategies for Ensuring Fairness in Machine Learning: With and Without Demographics

Towards Measuring Fairness in AI: the Casual Conversations Dataset

Is it Still Fair? A Comparative Evaluation of Fairness Algorithms through the Lens of Covariate Drift

Quantifying the Cross-sectoral Intersecting Discrepancies within Multiple Groups Using Latent Class Analysis Towards Fairness

Identifying and examining machine learning biases on Adult dataset

FairJob: A Real-World Dataset for Fairness in Online Systems

Data vs. Model Machine Learning Fairness Testing: An Empirical Study

Assessing fairness in machine learning models: A study of racial bias using matched counterparts in mortality prediction for patients with chronic diseases

Dataset Distribution Impacts Model Fairness: Single vs. Multi-Task Learning

Aleatoric and Epistemic Discrimination: Fundamental Limits of Fairness Interventions

The Impact of Data Preparation on the Fairness of Software Systems

50 Years of Test (Un)fairness: Lessons for Machine Learning

AIM: Attributing, Interpreting, Mitigating Data Unfairness

Retiring $Δ$DP: New Distribution-Level Metrics for Demographic Parity

Long-Term Fairness Inquiries and Pursuits in Machine Learning: A Survey of Notions, Methods, and Challenges

Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition

Bringing the People Back In: Contesting Benchmark Machine Learning Datasets