Abstract:Understanding deep models is crucial for deploying them in safety-critical applications. We introduce GIFT, a framework for deriving post-hoc, global, interpretable, and faithful textual explanations for vision classifiers. GIFT starts from local faithful visual counterfactual explanations and employs (vision) language models to translate those into global textual explanations. Crucially, GIFT provides a verification stage measuring the causal effect of the proposed explanations on the classifier decision. Through experiments across diverse datasets, including CLEVR, CelebA, and BDD, we demonstrate that GIFT effectively reveals meaningful insights, uncovering tasks, concepts, and biases used by deep vision classifiers. Our code, data, and models are released at <a class="link-external link-https" href="https://github.com/valeoai/GIFT" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the interpretability problem of the decision - making process of deep vision models (vision classifiers), especially in critical application scenarios. Specifically, the authors propose the GIFT framework to generate global, interpretable and faithful text explanations. These problems can be summarized as follows: 1. **Understanding the decisions of deep models**: - Deep learning models require highly reliable and interpretable decision - making mechanisms in critical applications such as self - driving cars and medical image analysis. - Users need clear and reliable explanations to build trust and ensure the effective deployment of these models. 2. **From local to global explanations**: - Existing explanation methods mostly focus on local explanations, that is, explaining the model decisions for specific input images, such as through saliency maps or counterfactual explanations. - Local explanations cannot provide a comprehensive understanding of the overall behavior of the model, while global explanations aim to provide an overall understanding of the model behavior. 3. **Interpretability and faithfulness of explanations**: - Explanations must be human - understandable, that is, intuitive and meaningful. - Explanations must also be faithful, that is, accurately reflect the decision - making process of the model and avoid misleading users. 4. **Automatically discover classification rules, fine - grained concepts and biases**: - Automatically identify the fine - grained relationships between the classification rules used by the classifier, data features and classification results, as well as potential biases. ### Main contributions of the GIFT framework - **Propose an automated framework for the first time**: It is used to generate global, faithful and explicitly interpretable explanations for visual classifiers. - **Combine local and global explanations**: Generate global explanations by collecting local counterfactual signals and using large language models (LLM) for reasoning. - **Verification tools**: Measure the relevance and causal association of explanations to ensure the faithfulness of explanations. - **Verify across multiple datasets**: Demonstrate the effectiveness of GIFT on different datasets, including CLEVR, CelebA and BDD - OIA. Through these contributions, the GIFT framework not only improves the interpretability of deep vision models, but also provides a powerful tool for understanding and improving these models.

GIFT: A Framework for Global Interpretable Faithful Textual Explanations of Vision Classifiers

Interpreting and Controlling Vision Foundation Models via Text Explanations

MindGPT: Interpreting What You See with Non-invasive Brain Recordings

GIST: Generating Image-Specific Text for Fine-grained Object Classification

GiT: Towards Generalist Vision Transformer through Universal Language Interface

DiG-IN: Diffusion Guidance for Investigating Networks -- Uncovering Classifier Differences Neuron Visualisations and Visual Counterfactual Explanations

Faithful Counterfactual Visual Explanations (FCVE)

MEGL: Multimodal Explanation-Guided Learning

GIFT: Learning Transformation-Invariant Dense Visual Descriptors Via Group CNNs

DeViL: Decoding Vision features into Language

TVE: Learning Meta-attribution for Transferable Vision Explainer

Probabilistic Conceptual Explainers: Trustworthy Conceptual Explanations for Vision Foundation Models

Visual Explanations of Image-Text Representations via Multi-Modal Information Bottleneck Attribution

Human-inspired Explanations for Vision Transformers and Convolutional Neural Networks

Fill in the blanks: Rethinking Interpretability in vision

Visual Explanations via Iterated Integrated Attributions

Relevant Irrelevance: Generating Alterfactual Explanations for Image Classifiers

VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks

Interpreting Object-level Foundation Models via Visual Precision Search

GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths

Diffexplainer: Towards Cross-modal Global Explanations with Diffusion Models