Abstract:Objective: Shapley additive explanations (SHAP) is a popular post-hoc technique for explaining black box models. While the impact of data imbalance on predictive models has been extensively studied, it remains largely unknown with respect to SHAP-based model explanations. This study sought to investigate the effects of data imbalance on SHAP explanations for deep learning models, and to propose a strategy to mitigate these effects. Materials and Methods: We propose to adjust class distributions in the background and explanation data in SHAP when explaining black box models. Our data balancing strategy is to compose background data and explanation data with an equal distribution of classes. To evaluate the effects of data adjustment on model explanation, we propose to use the beeswarm plot as a qualitative tool to identify "abnormal" explanation artifacts, and quantitatively test the consistency between variable importance and prediction power. We demonstrated our proposed approach in an empirical study that predicted inpatient mortality using the Medical Information Mart for Intensive Care (MIMIC-III) data and a multilayer perceptron. Results: Using the data balancing strategy would allow us to reduce the number of the artifacts in the beeswarm plot, thus mitigating the negative effects of data imbalance. Additionally, with the balancing strategy, the top-ranked variables from the corresponding importance ranking demonstrated improved discrimination power. Discussion and Conclusion: Our findings suggest that balanced background and explanation data could help reduce the noise in explanation results induced by skewed data distribution and improve the reliability of variable importance ranking. Furthermore, these balancing procedures improve the potential of SHAP in identifying patients with abnormal characteristics in clinical applications.

Shapley variable importance clouds for interpretable machine learning

Shapley variable importance cloud for interpretable machine learning

Shapley variable importance cloud for machine learning models

Variable importance analysis with interpretable machine learning for fair risk prediction

Variable Importance Clouds: A Way to Explore Variable Importance for the Set of Good Models

Shapley Marginal Surplus for Strong Models

Local Interpretable Model Agnostic Shap Explanations for machine learning models

Comparing interpretability and explainability for feature selection

Shapley Value: from Cooperative Game to Explainable Artificial Intelligence

Error Analysis of Shapley Value-Based Model Explanations: An Informative Perspective

A Unified Approach to Interpreting Model Predictions

L-shapley and c-shapley: Efficient model interpretation for structured data

Interpretability Latent Space Method: Exploiting Shapley Representation to Explain Latent Space

Manifold-based Shapley explanations for high dimensional correlated features

Enhancing the Interpretability of SHAP Values Using Large Language Models

An empirical study of the effect of background data size on the stability of SHapley Additive exPlanations (SHAP) for deep learning models

Combining Counterfactuals With Shapley Values To Explain Image Models

ShapG: new feature importance method based on the Shapley value

Helpful or Harmful Data? Fine-tuning-free Shapley Attribution for Explaining Language Model Predictions

Extracting spatial effects from machine learning model using local interpretation method: An example of SHAP and XGBoost

Balanced background and explanation data are needed in explaining deep learning models with SHAP: An empirical study on clinical decision making