Abstract:Convolutional Neural Networks (CNNs) have demonstrated outstanding performance in various domains, such as face recognition, object detection, and image segmentation. However, the lack of transparency and limited interpretability inherent in CNNs pose challenges in fields such as medical diagnosis, autonomous driving, finance, and military applications. Several studies have explored the interpretability of CNNs and proposed various post-hoc interpretable methods. The majority of these methods are feature-based, focusing on the influence of input variables on outputs. Few methods undertake the analysis of parameters in CNNs and their overall structure. To explore the structure of CNNs and intuitively comprehend the role of their internal parameters, we propose an Attribution Graph-based Interpretable method for CNNs (AGIC) which models the overall structure of CNNs as graphs and provides interpretability from global and local perspectives. The runtime parameters of CNNs and feature maps of each image sample are applied to construct attribution graphs (At-GCs), where the convolutional kernels are represented as nodes and the SHAP values between kernel outputs are assigned as edges. These At-GCs are then employed to pretrain a newly designed heterogeneous graph encoder based on Deep Graph Infomax (DGI). To comprehensively delve into the overall structure of CNNs, the pretrained encoder is used for two types of interpretable tasks: (1) a classifier is attached to the pretrained encoder for the classification of At-GCs, revealing the dependency of At-GC's topological characteristics on the image sample categories, and (2) a scoring aggregation (SA) network is constructed to assess the importance of each node in At-GCs, thus reflecting the relative importance of kernels in CNNs. The experimental results indicate that the topological characteristics of At-GC exhibit a dependency on the sample category used in its construction, which reveals that kernels in CNNs show distinct combined activation patterns for processing different image categories, meanwhile, the kernels that receive high scores from SA network are crucial for feature extraction, whereas low-scoring kernels can be pruned without affecting model performance, thereby enhancing the interpretability of CNNs.

Extraction of an Explanatory Graph to Interpret a CNN

Explanatory Graphs for CNNs.

Interpreting CNN Knowledge Via an Explanatory Graph.

Knowledge Via An Explanatory Graph

Exclusive Feature Constrained Class Activation Mapping for Better Visual Explanation.

A Pixel-Level Explainable Approach of Convolutional Neural Networks and Its Application

Unsupervised Learning of Neural Networks to Explain Neural Networks

Unsupervised Learning of Neural Networks to Explain Neural Networks (extended abstract)

An attribution graph-based interpretable method for CNNs

Interpretable CNNs for Object Classification

Inference Graphs for CNN Interpretation

Embedding deep networks into visual explanations

Learning local discrete features in explainable-by-design convolutional neural networks

Explainable Image Recognition with Graph-based Feature Extraction

GNNExplainer: Generating Explanations for Graph Neural Networks

Growing Interpretable Part Graphs on ConvNets via Multi-Shot Learning

Mining Object Parts from CNNs Via Active Question-Answering.

CNN-G: Convolutional Neural Network Combined With Graph for Image Segmentation With Theoretical Analysis

Training Interpretable Convolutional Neural Networks by Differentiating Class-specific Filters

Deeper Interpretability of Deep Networks

FACExplainer: Generating Model-faithful Explanations for Graph Neural Networks Guided by Spatial Information.