Abstract:Objective: Shapley additive explanations (SHAP) is a popular post-hoc technique for explaining black box models. While the impact of data imbalance on predictive models has been extensively studied, it remains largely unknown with respect to SHAP-based model explanations. This study sought to investigate the effects of data imbalance on SHAP explanations for deep learning models, and to propose a strategy to mitigate these effects. Materials and Methods: We propose to adjust class distributions in the background and explanation data in SHAP when explaining black box models. Our data balancing strategy is to compose background data and explanation data with an equal distribution of classes. To evaluate the effects of data adjustment on model explanation, we propose to use the beeswarm plot as a qualitative tool to identify "abnormal" explanation artifacts, and quantitatively test the consistency between variable importance and prediction power. We demonstrated our proposed approach in an empirical study that predicted inpatient mortality using the Medical Information Mart for Intensive Care (MIMIC-III) data and a multilayer perceptron. Results: Using the data balancing strategy would allow us to reduce the number of the artifacts in the beeswarm plot, thus mitigating the negative effects of data imbalance. Additionally, with the balancing strategy, the top-ranked variables from the corresponding importance ranking demonstrated improved discrimination power. Discussion and Conclusion: Our findings suggest that balanced background and explanation data could help reduce the noise in explanation results induced by skewed data distribution and improve the reliability of variable importance ranking. Furthermore, these balancing procedures improve the potential of SHAP in identifying patients with abnormal characteristics in clinical applications.

Cost‐sensitive tree SHAP for explaining cost‐sensitive tree‐based models

Ensembles of Random SHAPs

Explainable Adaptive Tree-based Model Selection for Time Series Forecasting

TsSHAP: Robust model agnostic feature-based explainability for time series forecasting

Beyond TreeSHAP: Efficient Computation of Any-Order Shapley Interactions for Tree Ensembles

Consistent Individualized Feature Attribution for Tree Ensembles

A cost sensitive decision tree algorithm with two adaptive mechanisms

Succint Interaction-Aware Explanations

Conditional expectation network for SHAP

A Human-Grounded Evaluation of SHAP for Alert Processing

Accurate and Intuitive Contextual Explanations using Linear Model Trees

Two Credit Scoring Models Based on Dual Strategy Ensemble Trees.

Approximating Score-based Explanation Techniques Using Conformal Regression

Optimal Counterfactual Explanations in Tree Ensembles

Feature-Specific Coefficients of Determination in Tree Ensembles

A comparative study of methods for estimating model-agnostic Shapley value explanations

A-PETE: Adaptive Prototype Explanations of Tree Ensembles

Balanced background and explanation data are needed in explaining deep learning models with SHAP: An empirical study on clinical decision making

Explaining the Model and Feature Dependencies by Decomposition of the Shapley Value

Cost-Time Sensitive Decision Tree With Missing Values

Cost-Aware Robust Tree Ensembles for Security Applications