Abstract:Transformer interpretability research is a hot topic in the area of deep learning. Traditional interpretation methods mostly use the final layer output of the Transformer encoder as masks to generate an explanation map. However, These approaches overlook two crucial aspects. At the coarse-grained level, the mask may contain uncertain information, including unreliable and incomplete object location data; at the fine-grained level, there is information loss on the mask, resulting in spatial noise and detail loss. To address these issues, in this paper, we propose a two-stage coarse-to-fine strategy (C2F-Explainer) for improving Transformer interpretability. Specifically, we first design a sequential three-way mask (S3WM) module to handle the problem of uncertain information at the coarse-grained level. This module uses sequential three-way decisions to process the mask, preventing uncertain information on the mask from impacting the interpretation results, thus obtaining coarse-grained interpretation results with accurate position. Second, to further reduce the impact of information loss at the fine-grained level, we devised an attention fusion (AF) module inspired by the fact that self-attention can capture global semantic information, AF aggregates the attention matrix to generate a cross-layer relation matrix, which is then used to optimize detailed information on the interpretation results and produce fine-grained interpretation results with clear and complete edges. Experimental results show that the proposed C2F-Explainer has good interpretation results on both natural and medical image datasets, and the mIoU is improved by 2.08% on the PASCAL VOC 2012 dataset.

Local Interpretation of Transformer Based on Linear Decomposition

Unwrapping The Black Box of Deep ReLU Networks: Interpretability, Diagnostics, and Simplification

Explaining the black-box model: A survey of local interpretation methods for deep neural networks

Explaining NonLinear Classification Decisions with Deep Taylor Decomposition

Interpretable Disentanglement of Neural Networks by Extracting Class-Specific Subnetwork

Better Explain Transformers by Illuminating Important Information

Hybrid CNN -Interpreter: Interpret local and global contexts for CNN-based Models

Functional Network: A Novel Framework for Interpretability of Deep Neural Networks

Neural Network Module Decomposition and Recomposition

Interpretable Neural Network Decoupling.

Local Interpretations for Explainable Natural Language Processing: A Survey

DecompX: Explaining Transformers Decisions by Propagating Token Decomposition

C2F-Explainer: Explaining Transformers Better Through a Coarse-to-Fine Strategy

Automated Natural Language Explanation of Deep Visual Neurons with Large Models

Interpreting Deep Neural Networks Through Model Transformation: Literature Review

Combining Transformers with Natural Language Explanations

How to Explain Neural Networks: A perspective of data space division

Interpreting Deep Learning Model Using Rule-based Method

A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts

Interpreting and Controlling Vision Foundation Models via Text Explanations