Abstract:There has been a significant surge of interest recently around the concept of explainable artificial intelligence (XAI), where the goal is to produce an interpretation for a decision made by a machine learning algorithm. Of particular interest is the interpretation of how deep neural networks make decisions, given the complexity and `black box' nature of such networks. Given the infancy of the field, there has been very limited exploration into the assessment of the performance of explainability methods, with most evaluations centered around subjective visual interpretation of the produced interpretations. In this study, we explore a more machine-centric strategy for quantifying the performance of explainability methods on deep neural networks via the notion of decision-making impact analysis. We introduce two quantitative performance metrics: i) Impact Score, which assesses the percentage of critical factors with either strong confidence reduction impact or decision changing impact, and ii) Impact Coverage, which assesses the percentage coverage of adversarially impacted factors in the input. A comprehensive analysis using this approach was conducted on several state-of-the-art explainability methods (LIME, SHAP, Expected Gradients, GSInquire) on a ResNet-50 deep convolutional neural network using a subset of ImageNet for the task of image classification. Experimental results show that the critical regions identified by LIME within the tested images had the lowest impact on the decision-making process of the network (~38%), with progressive increase in decision-making impact for SHAP (~44%), Expected Gradients (~51%), and GSInquire (~76%). While by no means perfect, the hope is that the proposed machine-centric strategy helps push the conversation forward towards better metrics for evaluating explainability methods and improve trust in deep neural networks.

Using SHAP to Measure Interpretability of Neuronal Feature Visualization.

ShapG: new feature importance method based on the Shapley value

SHAP values for Explaining CNN-based Text Classification Models

Manifold-based Shapley explanations for high dimensional correlated features

Shap-CAM: Visual Explanations for Convolutional Neural Networks Based on Shapley Value.

Understanding Neural Network Through Neuron Level Visualization.

Shapley variable importance cloud for interpretable machine learning

Combining Counterfactuals With Shapley Values To Explain Image Models

DeepCSHAP: Utilizing Shapley Values to Explain Deep Complex-Valued Neural Networks

Understanding contributing neurons via attribution visualization

Instance-wise or Class-wise? A Tale of Neighbor Shapley for Concept-based Explanation

Shapley variable importance clouds for interpretable machine learning

Explaining the Unexplained: Revealing Hidden Correlations for Better Interpretability

InterpretTime: a new approach for the systematic evaluation of neural-network interpretability in time series classification

How Well Do Feature-Additive Explainers Explain Feature-Additive Predictors?

A Causality-Erasing Baseline Values of SHAP Method for Enhancing Explainable Deep Networks in SAR Target Classification

Interpreting vision and language generative models with semantic visual priors

A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts

Enhancing the Interpretability of SHAP Values Using Large Language Models

Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithms

Identifying Interpretable Visual Features in Artificial and Biological Neural Systems