Abstract:There has been a significant surge of interest recently around the concept of explainable artificial intelligence (XAI), where the goal is to produce an interpretation for a decision made by a machine learning algorithm. Of particular interest is the interpretation of how deep neural networks make decisions, given the complexity and `black box' nature of such networks. Given the infancy of the field, there has been very limited exploration into the assessment of the performance of explainability methods, with most evaluations centered around subjective visual interpretation of the produced interpretations. In this study, we explore a more machine-centric strategy for quantifying the performance of explainability methods on deep neural networks via the notion of decision-making impact analysis. We introduce two quantitative performance metrics: i) Impact Score, which assesses the percentage of critical factors with either strong confidence reduction impact or decision changing impact, and ii) Impact Coverage, which assesses the percentage coverage of adversarially impacted factors in the input. A comprehensive analysis using this approach was conducted on several state-of-the-art explainability methods (LIME, SHAP, Expected Gradients, GSInquire) on a ResNet-50 deep convolutional neural network using a subset of ImageNet for the task of image classification. Experimental results show that the critical regions identified by LIME within the tested images had the lowest impact on the decision-making process of the network (~38%), with progressive increase in decision-making impact for SHAP (~44%), Expected Gradients (~51%), and GSInquire (~76%). While by no means perfect, the hope is that the proposed machine-centric strategy helps push the conversation forward towards better metrics for evaluating explainability methods and improve trust in deep neural networks.

VeriX: Towards Verified Explainability of Deep Neural Networks

Which Neural Network Makes More Explainable Decisions? an Approach Towards Measuring Explainability

Better Verified Explanations with Applications to Incorrectness and Out-of-Distribution Detection

Solving the enigma: Deriving optimal explanations of deep networks

From Robustness to Explainability and Back Again

Understanding the (Extra-)Ordinary: Validating Deep Model Decisions with Prototypical Concept-based Explanations

Can I Trust the Explainer? Verifying Post-hoc Explanatory Methods

ViCE: Visual Counterfactual Explanations for Machine Learning Models

Foiling Explanations in Deep Neural Networks

Improving VQA and its Explanations \\ by Comparing Competing Explanations

Explaining Explanations: An Overview of Interpretability of Machine Learning

Interpretable Deep Learning Models: Enhancing Transparency and Trustworthiness in Explainable AI

A Study on Multimodal and Interactive Explanations for Visual Question Answering

The Impact of Explanations on AI Competency Prediction in VQA

Less is More: Discovering Concise Network Explanations

Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces

A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts

Human-in-the-Loop Model Explanation via Verbatim Boundary Identification in Generated Neighborhoods

Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithms

Sim2Word: Explaining Similarity with Representative Attribute Words via Counterfactual Explanations

EXACT: Towards a platform for empirically benchmarking Machine Learning model explanation methods