On the mental ability of the dog.

J. R. Royce

DOI: https://doi.org/10.1126/SCIENCE.110.2868.666

IF: 56.9

1949-12-16

Science

Abstract:

What problem does this paper attempt to address?

Shapley variable importance clouds for interpretable machine learning

Yilin Ning,Marcus Eng Hock Ong,Bibhas Chakraborty,Benjamin Alan Goldstein,Daniel Shu Wei Ting,Roger Vaughan,Nan Liu

DOI: https://doi.org/10.48550/arXiv.2110.02484

2021-10-06

Abstract:Interpretable machine learning has been focusing on explaining final models that optimize performance. The current state-of-the-art is the Shapley additive explanations (SHAP) that locally explains variable impact on individual predictions, and it is recently extended for a global assessment across the dataset. Recently, Dong and Rudin proposed to extend the investigation to models from the same class as the final model that are "good enough", and identified a previous overclaim of variable importance based on a single model. However, this method does not directly integrate with existing Shapley-based interpretations. We close this gap by proposing a Shapley variable importance cloud that pools information across good models to avoid biased assessments in SHAP analyses of final models, and communicate the findings via novel visualizations. We demonstrate the additional insights gain compared to conventional explanations and Dong and Rudin's method using criminal justice and electronic medical records data.

Machine Learning,Human-Computer Interaction
' s response to reviews Title : Tumor Necrosis Factor-alpha Attenuates Starvation-Induced Apoptosis through Upregulation of Ferritin Heavy Chain in Hepatocellular Carcinoma Cells

Xing-rui Kou

Abstract:Xing R Kou (kouxingrui@gmail.com) Ying Y Jing (jingy4172@sohu.com) Wei J Deng (weijiedeng@cuhk.edu.hl) Kai Sun (zhesuk621@126.com) Zhi P Han (hanzhipeng0311@126.com) Fei Ye (yefei_5682@163.com) Guo F Yu (yugfeng@yeah.net) Qing M Fan (fanqm01@sina.com) Lu Gao (mouse.520@163.com) Qiu D Zhao (zhaoqd@gmail.com) Xue Zhao (xue51014@163.com) Rong Li (yolanda2158@yahoo.com.cn) Meng C Wu (wumengchao2012@126.com) Li X Wei (weilixin@yahoo.com)
Shapley Marginal Surplus for Strong Models

Daniel de Marchi,Michael Kosorok,Scott de Marchi

2024-08-17

Abstract:Shapley values have seen widespread use in machine learning as a way to explain model predictions and estimate the importance of covariates. Accurately explaining models is critical in real-world models to both aid in decision making and to infer the properties of the true data-generating process (DGP). In this paper, we demonstrate that while model-based Shapley values might be accurate explainers of model predictions, machine learning models themselves are often poor explainers of the DGP even if the model is highly accurate. Particularly in the presence of interrelated or noisy variables, the output of a highly predictive model may fail to account for these relationships. This implies explanations of a trained model's behavior may fail to provide meaningful insight into the DGP. In this paper we introduce a novel variable importance algorithm, Shapley Marginal Surplus for Strong Models, that samples the space of possible models to come up with an inferential measure of feature importance. We compare this method to other popular feature importance methods, both Shapley-based and non-Shapley based, and demonstrate significant outperformance in inferential capabilities relative to other methods.

Machine Learning
Variable Importance Clouds: A Way to Explore Variable Importance for the Set of Good Models

Jiayun Dong,Cynthia Rudin

DOI: https://doi.org/10.48550/arXiv.1901.03209

2020-02-10

Abstract:Variable importance is central to scientific studies, including the social sciences and causal inference, healthcare, and other domains. However, current notions of variable importance are often tied to a specific predictive model. This is problematic: what if there were multiple well-performing predictive models, and a specific variable is important to some of them and not to others? In that case, we may not be able to tell from a single well-performing model whether a variable is always important in predicting the outcome. Rather than depending on variable importance for a single predictive model, we would like to explore variable importance for all approximately-equally-accurate predictive models. This work introduces the concept of a variable importance cloud, which maps every variable to its importance for every good predictive model. We show properties of the variable importance cloud and draw connections to other areas of statistics. We introduce variable importance diagrams as a projection of the variable importance cloud into two dimensions for visualization purposes. Experiments with criminal justice, marketing data, and image classification tasks illustrate how variables can change dramatically in importance for approximately-equally-accurate predictive models

Machine Learning
Comparing interpretability and explainability for feature selection

Jack Dunn,Luca Mingardi,Ying Daisy Zhuo

DOI: https://doi.org/10.48550/arXiv.2105.05328

2021-05-12

Abstract:A common approach for feature selection is to examine the variable importance scores for a machine learning model, as a way to understand which features are the most relevant for making predictions. Given the significance of feature selection, it is crucial for the calculated importance scores to reflect reality. Falsely overestimating the importance of irrelevant features can lead to false discoveries, while underestimating importance of relevant features may lead us to discard important features, resulting in poor model performance. Additionally, black-box models like XGBoost provide state-of-the art predictive performance, but cannot be easily understood by humans, and thus we rely on variable importance scores or methods for explainability like SHAP to offer insight into their behavior. In this paper, we investigate the performance of variable importance as a feature selection method across various black-box and interpretable machine learning methods. We compare the ability of CART, Optimal Trees, XGBoost and SHAP to correctly identify the relevant subset of variables across a number of experiments. The results show that regardless of whether we use the native variable importance method or SHAP, XGBoost fails to clearly distinguish between relevant and irrelevant features. On the other hand, the interpretable methods are able to correctly and efficiently identify irrelevant features, and thus offer significantly better performance for feature selection.

Machine Learning
Local Interpretable Model Agnostic Shap Explanations for machine learning models

P. Sai Ram Aditya,Mayukha Pal

DOI: https://doi.org/10.48550/arXiv.2210.04533

2022-10-10

Abstract:With the advancement of technology for artificial intelligence (AI) based solutions and analytics compute engines, machine learning (ML) models are getting more complex day by day. Most of these models are generally used as a black box without user interpretability. Such complex ML models make it more difficult for people to understand or trust their predictions. There are variety of frameworks using explainable AI (XAI) methods to demonstrate explainability and interpretability of ML models to make their predictions more trustworthy. In this manuscript, we propose a methodology that we define as Local Interpretable Model Agnostic Shap Explanations (LIMASE). This proposed ML explanation technique uses Shapley values under the LIME paradigm to achieve the following (a) explain prediction of any model by using a locally faithful and interpretable decision tree model on which the Tree Explainer is used to calculate the shapley values and give visually interpretable explanations. (b) provide visually interpretable global explanations by plotting local explanations of several data points. (c) demonstrate solution for the submodular optimization problem. (d) also bring insight into regional interpretation e) faster computation compared to use of kernel explainer.

Machine Learning,Artificial Intelligence
A Unified Approach to Interpreting Model Predictions

Scott Lundberg,Su-In Lee

DOI: https://doi.org/10.48550/arXiv.1705.07874

2017-11-25

Abstract:Understanding why a model makes a certain prediction can be as crucial as the prediction's accuracy in many applications. However, the highest accuracy for large modern datasets is often achieved by complex models that even experts struggle to interpret, such as ensemble or deep learning models, creating a tension between accuracy and interpretability. In response, various methods have recently been proposed to help users interpret the predictions of complex models, but it is often unclear how these methods are related and when one method is preferable over another. To address this problem, we present a unified framework for interpreting predictions, SHAP (SHapley Additive exPlanations). SHAP assigns each feature an importance value for a particular prediction. Its novel components include: (1) the identification of a new class of additive feature importance measures, and (2) theoretical results showing there is a unique solution in this class with a set of desirable properties. The new class unifies six existing methods, notable because several recent methods in the class lack the proposed desirable properties. Based on insights from this unification, we present new methods that show improved computational performance and/or better consistency with human intuition than previous approaches.

Artificial Intelligence,Machine Learning
ShapG: new feature importance method based on the Shapley value

Chi Zhao,Jing Liu,Elena Parilina

2024-06-30

Abstract:With wide application of Artificial Intelligence (AI), it has become particularly important to make decisions of AI systems explainable and transparent. In this paper, we proposed a new Explainable Artificial Intelligence (XAI) method called ShapG (Explanations based on Shapley value for Graphs) for measuring feature importance. ShapG is a model-agnostic global explanation method. At the first stage, it defines an undirected graph based on the dataset, where nodes represent features and edges are added based on calculation of correlation coefficients between features. At the second stage, it calculates an approximated Shapley value by sampling the data taking into account this graph structure. The sampling approach of ShapG allows to calculate the importance of features efficiently, i.e. to reduce computational complexity. Comparison of ShapG with other existing XAI methods shows that it provides more accurate explanations for two examined datasets. We also compared other XAI methods developed based on cooperative game theory with ShapG in running time, and the results show that ShapG exhibits obvious advantages in its running time, which further proves efficiency of ShapG. In addition, extensive experiments demonstrate a wide range of applicability of the ShapG method for explaining complex models. We find ShapG an important tool in improving explainability and transparency of AI systems and believe it can be widely used in various fields.

Artificial Intelligence,Computer Science and Game Theory
Explaining black box decisions by Shapley cohort refinement

Masayoshi Mase,Art B. Owen,Benjamin Seiler

DOI: https://doi.org/10.48550/arXiv.1911.00467

IF: 5.414

2019-11-01

Machine Learning

Abstract:We introduce a variable importance measure to quantify the impact of individual input variables to a black box function. Our measure is based on the Shapley value from cooperative game theory. Many measures of variable importance operate by changing some predictor values with others held fixed, potentially creating unlikely or even logically impossible combinations. Our cohort Shapley measure uses only observed data points. Instead of changing the value of a predictor we include or exclude subjects similar to the target subject on that predictor to form a similarity cohort. Then we apply Shapley value to the cohort averages. We connect variable importance measures from explainable AI to function decompositions from global sensitivity analysis. We introduce a squared cohort Shapley value that splits previously studied Shapley effects over subjects, consistent with a Shapley axiom.
Error Analysis of Shapley Value-Based Model Explanations: An Informative Perspective

Ningsheng Zhao,Jia Yuan Yu,Krzysztof Dzieciolowski,Trang Bui

2024-05-30

Abstract:Shapley value attribution (SVA) is an increasingly popular explainable AI (XAI) method, which quantifies the contribution of each feature to the model's output. However, recent work has shown that most existing methods to implement SVAs have some drawbacks, resulting in biased or unreliable explanations that fail to correctly capture the true intrinsic relationships between features and model outputs. Moreover, the mechanism and consequences of these drawbacks have not been discussed systematically. In this paper, we propose a novel error theoretical analysis framework, in which the explanation errors of SVAs are decomposed into two components: observation bias and structural bias. We further clarify the underlying causes of these two biases and demonstrate that there is a trade-off between them. Based on this error analysis framework, we develop two novel concepts: over-informative and underinformative explanations. We demonstrate how these concepts can be effectively used to understand potential errors of existing SVA methods. In particular, for the widely deployed assumption-based SVAs, we find that they can easily be under-informative due to the distribution drift caused by distributional assumptions. We propose a measurement tool to quantify such a distribution drift. Finally, our experiments illustrate how different existing SVA methods can be over- or under-informative. Our work sheds light on how errors incur in the estimation of SVAs and encourages new less error-prone methods.

Artificial Intelligence,Machine Learning
Manifold-based Shapley explanations for high dimensional correlated features

Xuran Hu,Mingzhe Zhu,Zhenpeng Feng,Ljubiša Stanković

DOI: https://doi.org/10.1016/j.neunet.2024.106634

2024-08-14

Abstract:Explainable artificial intelligence (XAI) holds significant importance in enhancing the reliability and transparency of network decision-making. SHapley Additive exPlanations (SHAP) is a game-theoretic approach for network interpretation, attributing confidence to inputs features to measure their importance. However, SHAP often relies on a flawed assumption that the model's features are independent, leading to incorrect results when dealing with correlated features. In this paper, we introduce a novel manifold-based Shapley explanation method, termed Latent SHAP. Latent SHAP transforms high-dimensional data into low-dimensional manifolds to capture correlations among features. We compute Shapley values on the data manifold and devise three distinct gradient-based mapping methods to transfer them back to the high-dimensional space. Our primary objectives include: (1) correcting misinterpretations by SHAP in certain samples; (2) addressing the challenge of feature correlations in high-dimensional data interpretation; and (3) reducing algorithmic complexity through Manifold SHAP for application in complex network interpretations. Code is available at https://github.com/Teriri1999/Latent-SHAP.
An empirical study of the effect of background data size on the stability of SHapley Additive exPlanations (SHAP) for deep learning models

Han Yuan,Mingxuan Liu,Lican Kang,Chenkui Miao,Ying Wu

DOI: https://doi.org/10.48550/arXiv.2204.11351

IF: 5.414

2022-04-24

Machine Learning

Abstract:Nowadays, the interpretation of why a machine learning (ML) model makes certain inferences is as crucial as the accuracy of such inferences. Some ML models like the decision tree possess inherent interpretability that can be directly comprehended by humans. Others like artificial neural networks (ANN), however, rely on external methods to uncover the deduction mechanism. SHapley Additive exPlanations (SHAP) is one of such external methods, which requires a background dataset when interpreting ANNs. Generally, a background dataset consists of instances randomly sampled from the training dataset. However, the sampling size and its effect on SHAP remain to be unexplored. In our empirical study on the MIMIC-III dataset, we show that the two core explanations - SHAP values and variable rankings fluctuate when using different background datasets acquired from random sampling, indicating that users cannot unquestioningly trust the one-shot interpretation from SHAP. Luckily, such fluctuation decreases with the increase of the background dataset size. Also, we notice an U-shape in the stability assessment of SHAP variable rankings, demonstrating that SHAP is more reliable in ranking the most and least important variables compared to moderately important ones. Overall, our results suggest that users should take into account how background data affects SHAP results, with improved SHAP stability as the background sample size increases.
Explaining Predictive Uncertainty with Information Theoretic Shapley Values

David S. Watson,Joshua O'Hara,Niek Tax,Richard Mudd,Ido Guy

DOI: https://doi.org/10.48550/arXiv.2306.05724

2023-11-01

Abstract:Researchers in explainable artificial intelligence have developed numerous methods for helping users understand the predictions of complex supervised learning models. By contrast, explaining the $\textit{uncertainty}$ of model outputs has received relatively little attention. We adapt the popular Shapley value framework to explain various types of predictive uncertainty, quantifying each feature's contribution to the conditional entropy of individual model outputs. We consider games with modified characteristic functions and find deep connections between the resulting Shapley values and fundamental quantities from information theory and conditional independence testing. We outline inference procedures for finite sample error rate control with provable guarantees, and implement efficient algorithms that perform well in a range of experiments on real and simulated data. Our method has applications to covariate shift detection, active learning, feature selection, and active feature-value acquisition.

Machine Learning
Enhancing the Interpretability of SHAP Values Using Large Language Models

Xianlong Zeng

2024-08-24

Abstract:Model interpretability is crucial for understanding and trusting the decisions made by complex machine learning models, such as those built with XGBoost. SHAP (SHapley Additive exPlanations) values have become a popular tool for interpreting these models by attributing the output to individual features. However, the technical nature of SHAP explanations often limits their utility to researchers, leaving non-technical end-users struggling to understand the model's behavior. To address this challenge, we explore the use of Large Language Models (LLMs) to translate SHAP value outputs into plain language explanations that are more accessible to non-technical audiences. By applying a pre-trained LLM, we generate explanations that maintain the accuracy of SHAP values while significantly improving their clarity and usability for end users. Our results demonstrate that LLM-enhanced SHAP explanations provide a more intuitive understanding of model predictions, thereby enhancing the overall interpretability of machine learning models. Future work will explore further customization, multimodal explanations, and user feedback mechanisms to refine and expand the approach.

Human-Computer Interaction
From global to local MDI variable importances for random forests and when they are Shapley values

Antonio Sutera,Gilles Louppe,Van Anh Huynh-Thu,Louis Wehenkel,Pierre Geurts

DOI: https://doi.org/10.48550/arXiv.2111.02218

IF: 5.414

2021-11-03

Machine Learning

Abstract:Random forests have been widely used for their ability to provide so-called importance measures, which give insight at a global (per dataset) level on the relevance of input variables to predict a certain output. On the other hand, methods based on Shapley values have been introduced to refine the analysis of feature relevance in tree-based models to a local (per instance) level. In this context, we first show that the global Mean Decrease of Impurity (MDI) variable importance scores correspond to Shapley values under some conditions. Then, we derive a local MDI importance measure of variable relevance, which has a very natural connection with the global MDI measure and can be related to a new notion of local feature relevance. We further link local MDI importances with Shapley values and discuss them in the light of related measures from the literature. The measures are illustrated through experiments on several classification and regression problems.
Helpful or Harmful Data? Fine-tuning-free Shapley Attribution for Explaining Language Model Predictions

Jingtan Wang,Xiaoqiang Lin,Rui Qiao,Chuan-Sheng Foo,Bryan Kian Hsiang Low

2024-06-07

Abstract:The increasing complexity of foundational models underscores the necessity for explainability, particularly for fine-tuning, the most widely used training method for adapting models to downstream tasks. Instance attribution, one type of explanation, attributes the model prediction to each training example by an instance score. However, the robustness of instance scores, specifically towards dataset resampling, has been overlooked. To bridge this gap, we propose a notion of robustness on the sign of the instance score. We theoretically and empirically demonstrate that the popular leave-one-out-based methods lack robustness, while the Shapley value behaves significantly better, but at a higher computational cost. Accordingly, we introduce an efficient fine-tuning-free approximation of the Shapley value (FreeShap) for instance attribution based on the neural tangent kernel. We empirically demonstrate that FreeShap outperforms other methods for instance attribution and other data-centric applications such as data removal, data selection, and wrong label detection, and further generalize our scale to large language models (LLMs). Our code is available at <a class="link-external link-https" href="https://github.com/JTWang2000/FreeShap" rel="external noopener nofollow">this https URL</a>.

Machine Learning,Artificial Intelligence
Extracting spatial effects from machine learning model using local interpretation method: An example of SHAP and XGBoost

Ziqi Li

DOI: https://doi.org/10.1016/j.compenvurbsys.2022.101845

2022-09-01

Abstract:Machine learning and artificial intelligence (ML/AI), previously considered black box approaches, are becoming more interpretable, as a result of the recent advances in eXplainable AI (XAI). In particular, local interpretation methods such as SHAP (SHapley Additive exPlanations) offer the opportunity to flexibly model, interpret and visualise complex geographical phenomena and processes. In this paper, we use SHAP to interpret XGBoost (eXtreme Gradient Boosting) as an example to demonstrate how to extract spatial effects from machine learning models. We conduct simulation experiments that compare SHAP-explained XGBoost to Spatial Lag Model (SLM) and Multi-scale Geographically Weighted Regression (MGWR) at the parameter level. Results show that XGBoost estimates similar spatial effects as those in SLM and MGWR models. An empirical example of Chicago ride-hailing modelling is presented to demonstrate the utility of SHAP with real datasets. Examples and evidence in this paper suggest that locally interpreted machine learning models are good alternatives to spatial statistical models and perform better when complex spatial and non-spatial effects (e.g. non-linearities, interactions) co-exist and are unknown.

environmental studies,geography,regional & urban planning
Variational Shapley Network: A Probabilistic Approach to Self-Explaining Shapley values with Uncertainty Quantification

Mert Ketenci,Iñigo Urteaga,Victor Alfonso Rodriguez,Noémie Elhadad,Adler Perotte

2024-02-07

Abstract:Shapley values have emerged as a foundational tool in machine learning (ML) for elucidating model decision-making processes. Despite their widespread adoption and unique ability to satisfy essential explainability axioms, computational challenges persist in their estimation when ($i$) evaluating a model over all possible subset of input feature combinations, ($ii$) estimating model marginals, and ($iii$) addressing variability in explanations. We introduce a novel, self-explaining method that simplifies the computation of Shapley values significantly, requiring only a single forward pass. Recognizing the deterministic treatment of Shapley values as a limitation, we explore incorporating a probabilistic framework to capture the inherent uncertainty in explanations. Unlike alternatives, our technique does not rely directly on the observed data space to estimate marginals; instead, it uses adaptable baseline values derived from a latent, feature-specific embedding space, generated by a novel masked neural network architecture. Evaluations on simulated and real datasets underscore our technique's robust predictive and explanatory performance.

Machine Learning
A Comparative Analysis of Model Agnostic Techniques for Explainable Artificial Intelligence

Yifei Wang

DOI: https://doi.org/10.37256/rrcs.3220244750

2024-08-07

Abstract:Explainable Artificial Intelligence (XAI) has become essential as AI systems increasingly influence critical domains, demanding transparency for trust and validation. This paper presents a comparative analysis of prominent model agnostic techniques designed to provide interpretability irrespective of the underlying model architecture. We explore Local Interpretable Model-agnostic Explanations (LIME), SHapley Additive exPlanations (SHAP), Partial Dependence Plots (PDP), Individual Conditional Expectation (ICE) plots, and Anchors. Our analysis focuses on several criteria including interpretative clarity, computational efficiency, scalability, and user-friendliness. Results indicate significant differences in the applicability of each technique depending on the complexity and type of data, highlighting SHAP and LIME for their robustness and detailed output, whereas PDP and ICE are noted for their simplicity in usage and interpretation. The study emphasizes the importance of context in choosing appropriate XAI techniques and suggests directions for future research to enhance the efficacy of model agnostic approaches in explainability. This work contributes to a deeper understanding of how different XAI techniques can be effectively deployed in practice, guiding developers and researchers in making informed decisions about implementing AI transparency.
On the Failings of Shapley Values for Explainability

Xuanxiang Huang,Joao Marques-Silva

DOI: https://doi.org/10.1016/j.ijar.2023.109112

IF: 4.452

2024-01-11

International Journal of Approximate Reasoning

Abstract:Explainable Artificial Intelligence (XAI) is widely considered to be critical for building trust into the deployment of systems that integrate the use of machine learning (ML) models. For more than two decades Shapley values have been used as the theoretical underpinning for some methods of XAI, being commonly referred to as SHAP scores. Some of these methods of XAI now rank among the most widely used, including in high-risk domains. This paper proves that the existing definitions of SHAP scores will necessarily yield misleading information about the relative importance of features for predictions. The paper identifies a number of ways in which misleading information can be conveyed to human decision makers, and proves that there exist classifiers which will yield such misleading information. Furthermore, the paper offers empirical evidence that such theoretical limitations of SHAP scores are routinely observed in ML classifiers.

computer science, artificial intelligence

On the mental ability of the dog.

Shapley variable importance clouds for interpretable machine learning

' s response to reviews Title : Tumor Necrosis Factor-alpha Attenuates Starvation-Induced Apoptosis through Upregulation of Ferritin Heavy Chain in Hepatocellular Carcinoma Cells

Shapley Marginal Surplus for Strong Models

Variable Importance Clouds: A Way to Explore Variable Importance for the Set of Good Models

Comparing interpretability and explainability for feature selection

Local Interpretable Model Agnostic Shap Explanations for machine learning models

A Unified Approach to Interpreting Model Predictions

ShapG: new feature importance method based on the Shapley value

Explaining black box decisions by Shapley cohort refinement

Error Analysis of Shapley Value-Based Model Explanations: An Informative Perspective

Manifold-based Shapley explanations for high dimensional correlated features

An empirical study of the effect of background data size on the stability of SHapley Additive exPlanations (SHAP) for deep learning models

Explaining Predictive Uncertainty with Information Theoretic Shapley Values

Enhancing the Interpretability of SHAP Values Using Large Language Models

From global to local MDI variable importances for random forests and when they are Shapley values

Helpful or Harmful Data? Fine-tuning-free Shapley Attribution for Explaining Language Model Predictions

Extracting spatial effects from machine learning model using local interpretation method: An example of SHAP and XGBoost

Variational Shapley Network: A Probabilistic Approach to Self-Explaining Shapley values with Uncertainty Quantification

A Comparative Analysis of Model Agnostic Techniques for Explainable Artificial Intelligence

On the Failings of Shapley Values for Explainability