Abstract:Convolutional Neural Networks (CNNs) have demonstrated outstanding performance in various domains, such as face recognition, object detection, and image segmentation. However, the lack of transparency and limited interpretability inherent in CNNs pose challenges in fields such as medical diagnosis, autonomous driving, finance, and military applications. Several studies have explored the interpretability of CNNs and proposed various post-hoc interpretable methods. The majority of these methods are feature-based, focusing on the influence of input variables on outputs. Few methods undertake the analysis of parameters in CNNs and their overall structure. To explore the structure of CNNs and intuitively comprehend the role of their internal parameters, we propose an Attribution Graph-based Interpretable method for CNNs (AGIC) which models the overall structure of CNNs as graphs and provides interpretability from global and local perspectives. The runtime parameters of CNNs and feature maps of each image sample are applied to construct attribution graphs (At-GCs), where the convolutional kernels are represented as nodes and the SHAP values between kernel outputs are assigned as edges. These At-GCs are then employed to pretrain a newly designed heterogeneous graph encoder based on Deep Graph Infomax (DGI). To comprehensively delve into the overall structure of CNNs, the pretrained encoder is used for two types of interpretable tasks: (1) a classifier is attached to the pretrained encoder for the classification of At-GCs, revealing the dependency of At-GC's topological characteristics on the image sample categories, and (2) a scoring aggregation (SA) network is constructed to assess the importance of each node in At-GCs, thus reflecting the relative importance of kernels in CNNs. The experimental results indicate that the topological characteristics of At-GC exhibit a dependency on the sample category used in its construction, which reveals that kernels in CNNs show distinct combined activation patterns for processing different image categories, meanwhile, the kernels that receive high scores from SA network are crucial for feature extraction, whereas low-scoring kernels can be pruned without affecting model performance, thereby enhancing the interpretability of CNNs.

Interpreting CNN Knowledge Via an Explanatory Graph.

Knowledge Via An Explanatory Graph

Explanatory Graphs for CNNs.

Extraction of an Explanatory Graph to Interpret a CNN

A Pixel-Level Explainable Approach of Convolutional Neural Networks and Its Application

Unsupervised Learning of Neural Networks to Explain Neural Networks

Unsupervised Learning of Neural Networks to Explain Neural Networks (extended abstract)

Human-centric Transfer Learning Explanation Via Knowledge Graph [extended Abstract]

An attribution graph-based interpretable method for CNNs

Inference Graphs for CNN Interpretation

Growing Interpretable Part Graphs on ConvNets via Multi-Shot Learning

EGNN: Constructing explainable graph neural networks via knowledge distillation

Interpretable CNNs for Object Classification

Training Interpretable Convolutional Neural Networks by Differentiating Class-specific Filters

Interactively Transferring CNN Patterns for Part Localization

Learning FRAME Models Using CNN Filters for Knowledge Visualization.

GNNExplainer: Generating Explanations for Graph Neural Networks

Mining Object Parts from CNNs Via Active Question-Answering.

CNN-G: Convolutional Neural Network Combined With Graph for Image Segmentation With Theoretical Analysis

Towards Multi-Grained Explainability for Graph Neural Networks

Understanding CNN Hidden Neuron Activations Using Structured Background Knowledge and Deductive Reasoning