Abstract:We propose a technique for producing 'visual explanations' for decisions from a large class of Convolutional Neural Network (CNN)-based models, making them more transparent and explainable. Our approach—Gradient-weighted Class Activation Mapping (Grad-CAM), uses the gradients of any target concept (say 'dog' in a classification network or a sequence of words in captioning network) flowing into the final convolutional layer to produce a coarse localization map highlighting the important regions in the image for predicting the concept. Unlike previous approaches, Grad-CAM is applicable to a wide variety of CNN model-families: (1) CNNs with fully-connected layers (e.g.VGG), (2) CNNs used for structured outputs (e.g.captioning), (3) CNNs used in tasks with multi-modal inputs (e.g.visual question answering) or reinforcement learning, all without architectural changes or re-training. We combine Grad-CAM with existing fine-grained visualizations to create a high-resolution class-discriminative visualization, Guided Grad-CAM, and apply it to image classification, image captioning, and visual question answering (VQA) models, including ResNet-based architectures. In the context of image classification models, our visualizations (a) lend insights into failure modes of these models (showing that seemingly unreasonable predictions have reasonable explanations), (b) outperform previous methods on the ILSVRC-15 weakly-supervised localization task, (c) are robust to adversarial perturbations, (d) are more faithful to the underlying model, and (e) help achieve model generalization by identifying dataset bias. For image captioning and VQA, our visualizations show that even non-attention based models learn to localize discriminative regions of input image. We devise a way to identify important neurons through Grad-CAM and combine it with neuron names (Bau et al. in Computer vision and pattern recognition, 2017) to provide textual explanations for model decisions. Finally, we design and conduct human studies to measure if Grad-CAM explanations help users establish appropriate trust in predictions from deep networks and show that Grad-CAM helps untrained users successfully discern a 'stronger' deep network from a 'weaker' one even when both make identical predictions. Our code is available at <a href="https://github.com/ramprs/grad-cam/">https://github.com/ramprs/grad-cam/</a>, along with a demo on CloudCV (Agrawal et al., in: Mobile cloud visual media computing, pp 265–290. Springer, 2015) (<a href="http://gradcam.cloudcv.org">http://gradcam.cloudcv.org</a>) and a video at <a href="http://youtu.be/COjUB9Izk6E">http://youtu.be/COjUB9Izk6E</a>.

Explainable Image Similarity: Integrating Siamese Networks and Grad-CAM

ExplainLFS: Explaining neural architectures for similarity learning from local perturbations in the latent feature space

Sim2Word: Explaining Similarity with Representative Attribute Words via Counterfactual Explanations

Interpretable Network Visualizations: A Human-in-the-Loop Approach for Post-hoc Explainability of CNN-based Image Classification

A Novel Explainable Artificial Intelligence Model in Image Classification problem

Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization

Attributable Visual Similarity Learning

Axiomatic Explanations for Visual Search, Retrieval, and Similarity Learning

LICO: Explainable Models with Language-Image Consistency

Accurate Explanation Model for Image Classifiers using Class Association Embedding

Is Grad-CAM Explainable in Medical Images?

Exploring Simple Siamese Representation Learning

REVEL Framework to Measure Local Linear Explanations for Black-Box Models: Deep Learning Image Classification Case Study

Explaining Siamese networks in few-shot learning

Salience-CAM: Visual Explanations from Convolutional Neural Networks Via Salience Score

INSightR-Net: Interpretable Neural Network for Regression using Similarity-based Comparisons to Prototypical Examples

Cross-CAM: Focused Visual Explanations for Deep Convolutional Networks Via Training-Set Tracing

Explaining Deep Convolutional Neural Networks for Image Classification by Evolving Local Interpretable Model-agnostic Explanations

Beyond the Veil of Similarity: Quantifying Semantic Continuity in Explainable AI

Integrated Grad-CAM: Sensitivity-Aware Visual Explanation of Deep Convolutional Networks via Integrated Gradient-Based Scoring

SUNY: A Visual Interpretation Framework for Convolutional Neural Networks from a Necessary and Sufficient Perspective