Abstract:There has been a significant surge of interest recently around the concept of explainable artificial intelligence (XAI), where the goal is to produce an interpretation for a decision made by a machine learning algorithm. Of particular interest is the interpretation of how deep neural networks make decisions, given the complexity and `black box' nature of such networks. Given the infancy of the field, there has been very limited exploration into the assessment of the performance of explainability methods, with most evaluations centered around subjective visual interpretation of the produced interpretations. In this study, we explore a more machine-centric strategy for quantifying the performance of explainability methods on deep neural networks via the notion of decision-making impact analysis. We introduce two quantitative performance metrics: i) Impact Score, which assesses the percentage of critical factors with either strong confidence reduction impact or decision changing impact, and ii) Impact Coverage, which assesses the percentage coverage of adversarially impacted factors in the input. A comprehensive analysis using this approach was conducted on several state-of-the-art explainability methods (LIME, SHAP, Expected Gradients, GSInquire) on a ResNet-50 deep convolutional neural network using a subset of ImageNet for the task of image classification. Experimental results show that the critical regions identified by LIME within the tested images had the lowest impact on the decision-making process of the network (~38%), with progressive increase in decision-making impact for SHAP (~44%), Expected Gradients (~51%), and GSInquire (~76%). While by no means perfect, the hope is that the proposed machine-centric strategy helps push the conversation forward towards better metrics for evaluating explainability methods and improve trust in deep neural networks.

Explainability using Decision Trees and Monte Carlo Simulations

Which Neural Network Makes More Explainable Decisions? an Approach Towards Measuring Explainability

Transferring Black-Box Decision Making to a White-Box Model

Explain To Decide: A Human-Centric Review on the Role of Explainable Artificial Intelligence in AI-assisted Decision Making

Explaining the Predictions of Any Image Classifier via Decision Trees

A Survey of the Various Methodologies Towards making Artificial Intelligence More Explainable

Explainable Artificial Intelligence (XAI) to Enhance Trust Management in Intrusion Detection Systems Using Decision Tree Model

Structure and Reduction of MCTS for Explainable-AI

Importance measures derived from random forests: characterisation and extension

Hacking a surrogate model approach to XAI

Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithms

Case-based Explainability for Random Forest: Prototypes, Critics, Counter-factuals and Semi-factuals

Explainable Artificial Intelligence Approaches: A Survey

Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence

Algorithms for interpretable machine learning

Explainable artificial intelligence for Healthcare applications using Random Forest Classifier with LIME and SHAP

Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scary

Explaining Black-Box Algorithms Using Probabilistic Contrastive Counterfactuals

Helpful, Misleading or Confusing: How Humans Perceive Fundamental Building Blocks of Artificial Intelligence Explanations

Will Algorithms Blind People? The Effect of Explainable AI and Decision-Makers’ Experience on AI-supported Decision-Making in Government