Abstract:There has been a significant surge of interest recently around the concept of explainable artificial intelligence (XAI), where the goal is to produce an interpretation for a decision made by a machine learning algorithm. Of particular interest is the interpretation of how deep neural networks make decisions, given the complexity and `black box' nature of such networks. Given the infancy of the field, there has been very limited exploration into the assessment of the performance of explainability methods, with most evaluations centered around subjective visual interpretation of the produced interpretations. In this study, we explore a more machine-centric strategy for quantifying the performance of explainability methods on deep neural networks via the notion of decision-making impact analysis. We introduce two quantitative performance metrics: i) Impact Score, which assesses the percentage of critical factors with either strong confidence reduction impact or decision changing impact, and ii) Impact Coverage, which assesses the percentage coverage of adversarially impacted factors in the input. A comprehensive analysis using this approach was conducted on several state-of-the-art explainability methods (LIME, SHAP, Expected Gradients, GSInquire) on a ResNet-50 deep convolutional neural network using a subset of ImageNet for the task of image classification. Experimental results show that the critical regions identified by LIME within the tested images had the lowest impact on the decision-making process of the network (~38%), with progressive increase in decision-making impact for SHAP (~44%), Expected Gradients (~51%), and GSInquire (~76%). While by no means perfect, the hope is that the proposed machine-centric strategy helps push the conversation forward towards better metrics for evaluating explainability methods and improve trust in deep neural networks.

Machine Learning Interpretability: A Survey on Methods and Metrics

A Survey of Interpretable Machine Learning Methods

Explainable AI: A Review of Machine Learning Interpretability Methods

Interpretable and explainable machine learning: A methods‐centric overview with concrete examples

Quantifying Interpretability and Trust in Machine Learning Systems

A Unified Study of Machine Learning Explanation Evaluation Metrics

Interpretability and Explainability: A Machine Learning Zoo Mini-tour

A Survey of the Interpretability Aspect of Deep Learning Models

Review Study of Interpretation Methods for Future Interpretable Machine Learning

Interpretable machine learning: Fundamental principles and 10 grand challenges

Causal Interpretability for Machine Learning -- Problems, Methods and Evaluation

Interpretability of Machine Learning: Recent Advances and Future Prospects

Interpretability of deep learning models: A survey of results

Explaining Explanations: An Overview of Interpretability of Machine Learning

Towards A Rigorous Science of Interpretable Machine Learning

Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithms

Evaluating Explainability in Machine Learning Predictions through Explainer-Agnostic Metrics

Interpretability of Machine Learning Methods Applied to Neuroimaging

A Survey on the Explainability of Supervised Machine Learning

Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges